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ABSTRACT 


The  purpose  of  this  research  is  to  define  and  discuss  the  "best  value"  concept  and  hou 
it  relates  to  the  procurement  of  engineering  and  technical  ser\ices.  The  research  was 
conducted  by  a  review  of  regulations  and  policy  guidance,  interview  s  with  Government 
and  defense  industry-  personnel,  and  visits  to  Navy  Contracting  Acti\ities  that  procure 
engineering  and  technical  ser\  ices.  The  intent  of  the  study  was  to  analyze  how  the  "best 
value"  concept  is  currently  implemented,  problems  with  implementation,  and  to  offer 
recommendations  to  improve  best  value  procurements  of  engineering  and  technical 
services.  This  study  also  recommends  areas  for  future  study  that  may  further  improve  the 
use  of  "best  value"  procurements  for  professional  ser\'ices. 
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1.  INTRODUCTION 


A .  AREA  OF  RESEARCH 

Achieving  the  "best  value"  contract  is  increasingly  mentioned  as  an  objective  of 
Government  procurement  largely  due  to  the  many  stories  of  poor  performance  attributed  to 
contractors  selected  based  on  the  low  bid.  As  such,  the  "best  \'alue"  concept  is  receiving 
much  attention  regarding  its  definition  and  implementation  in  source  selection.  Achie^'ing 
the  "best  value"  is  based  on  reviewing  all  factors  including  price  before  making  a 
procurement  decision.  The  result  should  be  the  most  advantageous  procurement  for  the 
Government. 

Consumers  utilize  the  "best  value"  concept  on  a  daily  basis  when  making  value 
judgment  tradeoffs  betw'een  price  and  qualit)'.  Many  consumers  prefer  higher  quality  over 
price  while  others  prefer  a  lower  price  over  quality.  Depending  on  the  consumer,  either 
decision  could  equate  to  a  "best  \'alue"  judgment.  The  goal  of  "best  value"  in  Government 
procurement  is  in  determining  the  optimum  tradeoff  between  cost  and  quality  representing 
the  most  ad\  antageous  procurement  decision. 

"Best  value"  has  applications  extending  to  every  competitively  awarded  contract  for 
goods  and  services  procured  by  the  Department  of  Defense  (DoD)  and  is  not  without  its 
problems.  In  a  system  erected  on  the  foundation  of  full  and  open  competition,  not 
awarding  a  contract  to  an  offeror  who  is  technically  responsible  and  offering  the  low’est 
price  could  cause  considerable  protest,  especially  in  the  area  of  services  which  are 
traditionally  more  difficult  to  quantify.  "Best  value"  issues  revolve  around  areas  including; 
proper  acquisition  planning  for  a  "best  value"  procurement,  the  validity  of  technical  and 
cost  realism  evaluations,  and  how  to  equate  superiority  of  one  contractor's  technical 
approach  with  the  low  er  bid  of  another  contractor.  From  a  contractor's  standpoint,  with  a 


1 


decreasing  number  of  contracts  available,  there  exists  a  high  le\  el  of  distrust  in  the  "best 
value"  source  selection  process.  Many  contractors  believe  that,  no  matter  what  the 
Government  says,  the  competition  is  based  on  low  bid.  This  thesis  research  will  examine 
the  "best  value"  concept  in  source  selection  for  engineering  and  technical  sei^  ices  contracts. 
The  objecti\e  is  to  analyze  the  current  use  of  the  "best  value"  concept  and  make 
recommendations  for  impro\  ement  of  the  process. 

This  research  was  accomplished  through  personal  and  telephone  interviews  with 
Government  and  defense  contractor  personnel.  In  addition  to  the  inter\  ie\\  s,  examples  of 
procurement  documentation  were  collected  and  analyzed. 


B.  RESEARCH  QUESTIONS 

1 .  Primary  Research  Question 

What  is  the  most  efficient  and  effective  method  of  implementing  "best  value"  in 
source  selection  for  engineering  and  technical  ser%  ices  contracts? 

2 .  Subsidiary  Research  Questions 

1 .  What  is  "best  value"? 

2.  Under  what  circumstances  is  "best  value"  employed? 

3 .  What  are  the  "best  N  alue"  source  selection  techniques  for  engineering  and  technical 
services? 

4.  What  factors  and  subfactors  are  utilized  in  "best  value"  source  selections  for 
engineering  and  technical  sen  ice  contracts? 

5.  What  are  the  problems  or  impediments  to  implementing  "best  value"  source 
selection  for  engineering  and  technical  sen  ices? 

6.  How  do  contracting  activities  quantify  paying  a  premium  on  "best  value" 
procurements? 


C.  OBJECTIVE 

The  objective  of  this  research  is  to  examine  the  intricacies  and  problems  associated 
with  the  use  of  the  "best  value"  concept  in  source  selection.  By  doing  so,  qualitative 
insight  will  be  provided  to  procurement  personnel  and  engineering  support  services 


contractors. 


D.  SCOPE  OF  THESIS 

The  scope  of  this  thesis  revolved  around  the  acquisition  planning  and  evaluation 
phases  of  the  contracting  cycle  for  engineering  and  technical  serv  ices  contracts.  Contract 
administration  is  only  briefly  mentioned  in  the  conte.xt  of  viewpoints  collected  during 
research  on  a  specific  problem;  contractor  gaming,  which  is  discussed  in  Chapter  IV.  The 
research  focused  on  Navy  contracting  activities  although  other  DoD  agencies  and  the 
National  Aeronautics  and  Space  Administration  (NASA)  w  ere  contacted  for  their  positions 
in  se\eral  areas.  B\'  contacting  other  agencies,  the  researcher  was  able  to  compare 
problems  and  solutions  to  the  "best  value"  approach.  Similarities  between  systems 
commands  and  field  acti\  ities  were  expected. 

E.  METHODOLOGY 

The  methodology  used  in  this  study  consisted  of  a  comprehensive  literature  re\  iew , 
personal,  and  telephone  interviews,  and  onsite  visits  to  contracting  activities.  This 
methodology  was  necessary'  because  the  "best  value"  concept  is  somewhat  nebulous,  and 
as  a  result,  interpersonal  communications  were  required.  A  sample  of  eight  to  ten 
engineering  senices  contracts  was  reviewed  in  the  stud\.  Government  acquisition 
personnel,  defense  contractors,  and  requiring  activities  were  targeted  as  the  sources  of 
information  for  the  study. 

F.  ORGANIZATION 

Chapter  I  defined  the  research  issue  and  its  importance.  The  objecti\e,  scope,  and 
methodology  of  the  thesis  w'ere  also  presented. 
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Chapter  II  examines  a  definition  of  the  "best  \  alue"  concept  presented  in  the  literature 
and  discusses  when  "best  value"  should  be  utilized.  In  addition,  the  chapter  defines 
engineering  and  technical  serx  ices  and  "best  value"  source  selection  methods  used  to  obtain 
these  serx'ices. 

Chapter  III  examines  responses  received  from  a  suix  ey  of  Gox  emment  and  contractor 
personnel.  In  addition,  data  obtained  from  a  sample  of  engineering  and  technical  serx  ices 
contracts  is  presented. 

Chapter  IV  presents  an  analysis  of  problems  collected  from  the  interviews  with 
Goxemment  and  contractor  personnel. 

Chapter  V  provides  conclusions  and  recommendations  germane  to  the  research 
conducted.  Additionally,  the  research  questions  are  answered  and  areas  for  further 
research  are  presented. 
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II.  BACKGROUND 


A.  INTRODUCTION 

This  chapter  will  define  the  "best  value"  concept  and  discuss  when  "best  value"  is 
utilized.  This  chapter  also  introduces  a  definition  of  engineering  and  technical  ser\’ices,  and 
a  discussion  of  "best  value"  source  selection  methods  used  to  procure  engineering  and 
technical  ser\  ices. 

B.  "BEST  VALUE"  DEFINED 

Examples  of  definitions  for  "best  value"  from  a  literature  search  are  presented  below. 

In  a  report  on  expanding  the  use  of  commercial  components  in  militaiA’  equipment,  the 
Defense  Science  Board  offered  the  following  definition: 

"Best  value"— the  evaluation  of  a  (commercial)  product,  system  or  serx  ice  based  on 
all  reasonable  factors  including  but  not  limited  to  initial  price,  life  cycle  costs,  available 
extended  warranties,  prior  product  experience,  availability  of  distribution  and  sen  ice 
channels,  past  producer  performance,  past  vendor  performance  and  so  forth,  for  the 
purpose  of  procuring  a  product,  system  or  serx  ice  that  prox  ides  optimum  satisfaction  of 
the  mission  needs.  [Ref.  l:p.  81] 

The  Nax  al  Supply  Systems  Command  offers  a  similar  definition  by  describing  "best 
value"  as  a  process  xx  hich,  "Describes  all  those  acquisitions  xvhere  selection  for  axvard  is  to 
be  based  on  other  than  the  lowest  exaluated  price  or  lowest  cost/price  among  technically 
acceptable  offers."  [Ref.  2;p.  A-4] 

The  Naval  Air  Systems  Command  (NAVAIR)  is  more  succinct  xx  ith  its  definition  of 
"best  value"  which  encompasses  not  only  the  process  of  proposal  evaluation  on  factors 
other  than  price,  but  also  incorporates  the  definition  of  the  requirement  as  xvell.  In  a  1990 
Business  Strategy  Presentation,  NAVAIR  defined  "best  xalue"  as  "obtaining  the  right 
product  at  the  right  price,  and  not  making  an  acquisition  decision  on  price  alone." 
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The  Defense  Logistic  Agency  (DLA),  in  a  handbook  for  contracting  officers,  DLA 
Handbook  4105.3  [Ref.  3],  is  more  expansive  in  its  definition  of  "best  value".  It  considers 
the  usage  of  value  judgments  for  all  methods  of  source  selection.  These  value  judgments 
are  employed  as  methods  of  achieving  the  most  advantageous  procurement  decision.  For 
instance,  in  a  source  selection  based  on  low  price,  the  contracting  officer  should  consider 
additional  costs  to  the  Government  of  quality  deficiency  reports  and  delinquency  on  past 
contracts  incurred  by  the  offeror  being  evaluated.  These  added  costs  should  be  factored  in 
as  expected  costs  prior  to  making  an  award  decision.  Buying  the  "best  value"  in  this 
context  refers  to  awarding  a  contract  on  the  low  overall  cost  instead  of  low  initial  price. 
DLAH  4105.3  recommends  techniques  to  obtain  the  objective  of  "best  value"  for  various 
methods  of  source  selection  including  formal  source  selection,  streamlined  source 
selection,  responsibilit}’  determinations,  quality  vendors,  and  multi-source  contracting. 

The  Defense  Personnel  Support  Center  (DPSC)  in  a  handbook  or.  source  selection, 
DPSC  Handbook  4105.8,  describes  "best  value"  as  a  source  selection  technique  which 
equates  to  a  streamlined  \  ersion  of  formal  source  selection.  "Best  \  alue"  source  selection 
in  this  context,  is  utilized  for  less  complex  procurements  with  dollar  values  under  $1 
million  [Ref.  4:p.  7]. 

The  Defense  Analysis  and  Studies  Office  also  viewed  "best  value"  as  a  variation  on 
source  selection  when  it  reported; 

One  variation  on  source  selection,  "Best  Value"  Source  Selection  (BVSS), 
includes  the  same  elements  used  in  the  usual  source  selection  process.  It  is 
streamlined,  howe\er,  by  lowering  the  approval  signature  levels  and  by  simplifying 
some  of  the  procedures  BVSS  also  places  emphasis  on  contractor  past  performance  in 
both  quality  and  on  time  deli\'ery.  [Ref.  5;p.  5] 

The  Air  Force,  Army,  and  NASA  utilize  source  selection  methods  based  on  factors 
other  than  price  alone,  yet  the  term  "best  value",  "best  buy"  or  "greatest  value"  could  not  be 
located  in  their  source  selection  literature.  General  references  allude  to  source  selection 
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methods  \\  hich  "allo\\  an  award  to  be  based  on  factors  other  than  price"  [Ref.  6;p.  2]  and 
"selecting  the  source  whose  proposal  presents  the  highest  probability  of  quality 
performance  to  best  meet  NASA's  requirements  at  a  realistic  cost/price"  [Ref.  7:p.  i]. 
Howe\er,  none  mention  "best  value"  as  a  specific  type  of  procurement  process. 
References  to  "best  value",  greatest  \alue,  and  greatest  overall  benefit  to  the  Government 
are  made  in  sample  Request  For  Propxjsals  (RFP)s  from  all  DoD  agencies,  further 
indicating  that  all  agencies  mentioned  do  have  an  appreciation  for  the  concept. 

For  purposes  of  this  research,  a  working  definition  for  "best  value"  is  presented  in 
terms  of  an  objective  and  a  process.  As  an  objective,  "best  value"  represents  the  most 
advantageous  procurement  decision  as  ailable  to  the  Government.  This  decision  is  based  on 
low  price,  low  os  erall  cost  or  other  factors.  "Best  value"  defined  as  an  objective  has 
extremely  broad  applications.  One  can  obtain  the  "best  value"  under  any  source  selection 
method  as  long  as  the  product  or  serv  ice  meets  the  criterion  of  providing  optimum 
satisfaction  of  mission  needs. 

"Best  value"  is  also  defined  in  terms  of  an  evaluation  process  where  all  rele\  ant  factors 
are  taken  into  account  prior  to  making  a  procurement  decision.  The  end  result  of  this 
evaluation  process  represents  the  most  advantageous  procurement  decision.  In  this 
context,  procurement  decisions  are  not  based  on  price  alone. 

As  an  evaluation  process,  "best  \  alue"  directly  contrasts  with  technically  acceptable 
low  offeror  evaluations  which  make  contract  awards  based  on  the  low  price  among 
technically  acceptable  offerors.  "Best  value"  emphasizes  other  factors  in  addition  to  price 
and  establishes  different  levels  of  technical  merit  from  which  to  make  the  most 
advantageous  procurement  decision. 
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C.  FAR  AND  DFARS  GUIDANCE  ON  "BEST  VALUE 


A  re\  ie\\  of  the  FAR  and  DFARS  for  the  term  "best  (greatest)  value"  unco\  ers  \  er\ 
few  specifics,  although  factors  that  imply  the  use  of  the  "best  value"  concept  of  making  the 
most  ad\antageous  procurement  decision  based  on  price  and  other  factors  are  implicit 
throughout. 

The  only  section  of  FAR  within  which  the  "best  value"  concept  is  specifically  referred 

to  is  in  the  source  selection  subpart  15.605,  which  states: 

Where  the  lowest  price  or  lowest  cost  to  the  Government  is  properly  the  deciding 
factor  in  many  source  selections,  in  certain  acquisitions,  the  Government  may  select  the 
source  whose  proposal  offers  the  greatest  value  to  the  Government  in  terms  of 
performance  and  other  factors.  [Ref.  8:p.  15-20] 

The  FAR  continues  its  reference  to  "best  value"  as  follows: 

In  awarding  a  cost  reimbursement  contract,  the  cost  proposal  should  not  be 
controlling  since  ad\’ance  estimates  of  costs  may  not  be  \  alid  indicators  of  actual  costs. 
The  primary  consideration  should  be,  which  offeror  can  perform  the  contract  in  a 
manner  most  advantageous  to  the  Government  as  determined  by  evaluation  of 
proposals  according  to  the  established  evaluation  criteria.  [Ref.  8:p.  15-20] 

A  recent  change  to  the  DFARS,  published  in  the  Federal  Register,  Vol.  56,  No.  172, 
page  43986,  on  September  5,  1991,  is  more  explicit  with  regard  to  "best  value"  in  the  area 
of  services.  The  change  is  a  result  of  Section  834  of  the  1991  National  Defense 
Authorization  Act  which  requires  DoD,  to  the  maximum  extent  possible,  to  acquire  services 
on  the  basis  of  tasks  performed  rather  than  on  the  basis  of  number  of  hours  the  service  is  to 
be  provided.  DFARS  section  215.605  is  amended  to  read  as  follows: 

215.605  Evaluation  Factors 

(C)  In  Competitive  acquisitions  of  sei^  ices— 

( 1)  Evaluation  and  award  should  be  based  to  the  maximum  extent  practicable,  on  best 
overall  value  to  the  Go\'emment  in  terms  of  quality  and  other  factors. 

(2)  The  weighting  of  costs  must  be  commensurate  with  the  nature  of  the  services 
provided 
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The  change  continues  by  delineating  when  it  might  be  appropriate  to  use  low  price 
technicalh’  acceptable  offeror,  and  \\'hen  it  might  be  more  appropriate  to  aw  ard  to  an  offeror 
based  on  technical  and  quality  considerations  at  other  than  the  lowest  price.  The  DFARS 
change  also  adds  section  237.170  which  provides  guidance  and  prescribes  a  new 
solicitation  prov  ision  and  contract  clause  on  uncompensated  overtime. 


D.  WHEN  IS  "BEST  VALUE"  UTILIZED? 

The  use  of  "best  value"  is  generally  reserved  for  the  procurement  of  comple.x  goods  or 
services  where  alternate  approaches  for  solving  a  requirement  are  available.  This  is  due  to 
the  often  times  subjective  nature  of  a  "best  value"  procurement  where  evaluation  factors  and 
criteria  within  those  factors  are  utilized  to  discriminate  between  solutions  offered  by 
competing  proposals.  As  a  result,  requirements  necessitating  the  use  of  performance 
specifications  or  complex  services  are  typically  considered  for  "best  value"  evaluations. 
Conversely,  products  with  detailed  sp)ccifications  and  routine  services  are  generally 
acquired  based  on  low  cost. 

In  a  memorandum  to  program  executive  officers,  direct  reporting  program  managers, 
commanders  of  system  commands,  and  others,  the  Assistant  Secretary  of  the  Navy  for 
Research,  Development  and  Acquisition,  the  Honorable  Gerald  Cann  described  using  "best 
value"  as  follows; 

"Best  Value"  evaluations  should  be  employed  whenever  possible  and  especially  in 
weapons  system,  federal  information  processing  and  professional  and  technical  service 
support  acquisitions.  Requirements  which  dictate  complex  integration  of  people, 
equipment,  hardware,  innovation  and  software  should  also  be  evaluated  on  a  "best 
value"  basis.  However,  when  non-complex,  routine  requirements  are  being  procured 
and  the  product  to  be  deliv  ered  is  clearly  defined  at  the  outset  of  the  procurement,  it 
ma>’  be  appropriate  to  award  to  the  lowest  priced,  technically  acceptable  offeror.  [Ref. 

9,  Appendix  A] 
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Due  lo  Ihc  non-definilive  nature  of  ’’best  value"  procurements,  contract  type  selection 
is  normally  cost  reimbursement.  This  is  not  to  say  that  fixed  price  arrangements  are  not 
used  for  "best  value"  procurements.  Ho\\'ever, 

Great  care  is  required  in  the  application  of  greatest  (best)  value  to  fixed  price  efforts 
since  the  application  of  e\  aluation  factors  to  distinguish  between  offers  for  an  item 
which  has  an  adequately  detailed  specification  may  involve  applying  responsibility 
factors  rather  than  technical  acceptability  factors.  [Ref.  2:p.  A-4j 

The  point  is  that  the  contracting  activity  cannot  eliminate  an  offeror,  especially  a  small 

business  for  responsibility  type  issues  through  the  use  of  evaluation  factors.  The 

responsibility  determination  is  made  after  the  "best  value"  offeror  is  selected. 

Comptroller  General  decision  B-238953.4  of  28  September  1990,  in  the  matter  of 
night  International  Group,  Inc.  (FLIGHT),  may  be  instructive  concerning  the  issue  of 
responsibility  determination  in  lieu  of  "best  value"  evaluations.  In  this  case,  the  Na\’y  used 
a  responsibility  factor  to  eliminate  a  small  business  under  the  guise  of  a  "best  value" 
evaluation. 

FLIGHT  challenged  the  Navy's  evaluation  of  its  proposal  because  the  financial 
condition  of  the  company  was  not  stated  as  an  evaluation  factor  in  the  solicitation.  The 
solicitation  requested  proposals  for  a  firm  fixed  price  contract  to  provide  flight  training 
services  for  Naval  Right  Officers  for  one  base  year  and  four  option  years.  The  solicitation 
provided  for  an  award  to  the  offeror  whose  proposal  presented  the  "best  \’aluc"  to  the 
Government  on  the  basis  of  three  factors;  technical  approach,  and  price,  which  w'ere 
weighted  equally,  and  management,  which  was  considerably  less  important.  During 
proposal  evaluation,  it  w'as  learned  through  a  newspajjer  article  that  FLIGHT,  a  small 
business,  was  having  financial  difficulties  and  was  "considering  options  to  meet  liquidity 
problems."  The  contracting  officer  contacted  the  cognizant  Defense  Contract 
Administration  Services  Management  Area  (DCASMA)  which  recommended  against  the 
award.  The  contracting  officer  then  failed  to  contact  the  Small  Business  Administration  for 


10 


a  required  certificate  of  competency  determination  for  FLIGHT.  This  was  due  to  the 
pressure  of  a  time  constraint  to  award  the  contract.  The  result  was  an  e\aluation  of 
FLIGHT'S  proposal: 

On  the  basis  of  inadequate  financial  capacity  but  under  the  guise  of  a  comparati\  c 
"best  value"  evaluation  (which)  effectively  constituted  a  finding  of  nonresponsibility 
which  the  Agenev  was  required  to  refer  to  the  Small  Business  Administration.  [Ref. 

10:p.  2] 

The  protest  by  FLIGHT  was  sustained.  The  Navy  was  ordered  to  terminate  the 
current  contract  for  convenience  and  award  the  contract  to  FLIGHT.  In  the  FT^IGHT  case, 
the  Navy  employed  a  responsibility  type  factor  to  eliminate  a  small  business  in  lieu  of  a 
certificate  of  competency  determination  from  the  Small  Business  Administration. 

E.  WHAT  ARE  ENGINEERING  AND  TECHNICAL  SERVICES  AND 

WHEN  ARE  THEY  USED? 

Service  contracting  characteristics  and  governing  regulations  are  described  in  FAR  part 
37  which  defines  a  service  contract  in  subpart  37. 101  as: 

A  contract  that  directly  engages  the  time  and  effort  of  a  contractor  whose  primarv’ 
purpose  is  to  perform  an  identifiable  task  rather  than  furnish  an  end  item  of  supply.  A 
sen  ice  contract  may  be  a  non-personal  or  personal  contract.  It  can  also  cov  er  sen  ices 
performed  by  either  professional  or  non-professional  personnel  whether  on  an 
individual  or  organizational  basis.  [Ref.  8:p.  37-1] 

The  DoD  awards  sen  ice  contracts  for  efforts  encompassing  routine,  recurring  type  sen  ices 

to  sophisticated  research  and  design  efforts. 

As  delineated  in  the  FAR,  part  37,  engineering  and  technical  senices  are  one  type  of 
Contracted  Advisory  and  Assistance  Services  (CAAS)  which  take  the  form  of 
"Information,  advice,  opinions,  alternatives,  conclusions,  recommendations,  training  or 
direct  assistance"  [Ref,  8:p.  37-3].  Besides  engineering  and  technical  senices,  other 
senices  covered  under  the  CAAS  genre  include  individual  experts  and  consultants,  studies, 
analyses  and  evaluations,  and  management  and  professional  support  senices. 
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Contractor  engineering  and  technical  serv  ices  are  defined  in  DFARS  237.701  a.s; 

The  furnishing  of  advice,  instruction,  and  training  to  DoD  personnel,  by 
commercial  or  industrial  companies  in  the  installation,  operation,  and  maintenance  of 
DoD  weapons,  equipment  and  systems.  This  includes  transmitting  the  knowledge 
necessary  to  de\elop  among  those  DoD  personnel  the  technical  skill  required  for 
installing,  maintaining  and  operating  such  equipment  in  a  high  state  of  military 
readiness.  [Ref.  ll:p.  237-6] 

DFARS  further  separates  engineering  and  technical  ser\’ices  into  three  categories  which 
include  contract  field  sen  ices,  contract  plant  ser\  ices,  and  field  ser\  ice  representatix  es. 

1 .  Contracting  for  Services 

Service  contracting  is  differentiated  from  hardware  procurement  in  that  with 
serv  ices,  the  Government  is  acquiring  expertise,  knowledge,  and  time  instead  of  an  end 
item  of  supply.  As  a  result,  the  process  of  acquiring  serv  ices  is  somewhat  different. 

In  acquiring  services,  the  requiring  activity  must  first  make  a  determination  that  the 
services  cannot  be  performed  "in  house";  meaning  Government  employees  have  neither  the 
expertise  nor  the  resources  to  perform  the  requirement.  Once  this  determination  is  made, 
the  requiring  activity  constructs  a  Performance  Work  Statement  (PWS)  in  lieu  of  a 
specification  for  hardware  requirements.  The  PWS  is  a  document  which  describes  the 
essential  and  technical  requirements  for  the  sen  ice  and  the  standards  used  to  determine 
whether  the  requirements  are  met.  The  PWS  for  engineering  and  technical  service  contracts 
requiring  "best  value"  source  selection  is  usually  broad  in  scope  providing  the  flexibility 
necessary  to  meet  the  needs  of  dynamic  programs.  This  is  in  contrast  to  well  defined 
PWSs  for  routine  services  and  design  specifications  for  hardware  procurements. 

Contracting  for  services  employs  a  different  set  of  evaluation  criteria  than 
contracting  for  hardware  and  other  tangible  items.  Since  technical  expertise  is  of 
paramount  importance  with  services,  technical  factors  are  heav  ily  weighted.  These  include 
the  contractor's  technical  approach  to  solving  the  requirement,  the  level  of  experience 
among  key  personnel  and  corporate  experience  in  similar  or  past  efforts. 


Solicitations  for  engineering  and  technical  service  contracts  will  normally  include 
an  estimated  ceiling  co^■ering  the  number  of  hours  during  a  base  year  and  subsequent 
option  years,  and  the  labor  categories,  or  "labor  mix,"  necessar}  to  complete  the  tasks 
presented  in  the  PWS.  Categories  in  the  labor  mix  are  sometimes  equated  to  an  equivalent 
Government  sen  ice  rating,  supplying  the  offeror  with  the  minimum  pay  scale  expected. 
These  rates  are  then  used  as  benchmarks  to  evaluate  the  cost  realism  of  the  offeror's 
proposal.  The  cost  realism  analysis  is  discussed  further  in  the  next  section  of  this  chapter. 

Engineering  and  technical  services  are  procured  by  field  contracting  activities, 
laboratories,  weapon  testing  sites,  training  centers  and  systems  commands.  Examples  of 
engineering  and  technical  ser\  ices  encountered  by  this  researcher  include  configuration 
management  support,  systems  and  software  engineering  support,  ser\’ices  in  support  of 
weapons  lethality  assessment,  and  cruise  missile  technical  support  seix  ices  among  others. 

Contracts  for  engineering  and  technical  seix'ices  usually  consist  of  one  base  year 
with  the  option  to  continue  senices  for  up  to  four  additional  years.  Therefore,  if  all 
options  are  exercised,  the  maximum  length  of  a  contract  can  extend  to  five  years  prior  to 
resoliciting  the  requirement.  The  contracts  can  be  omnibus  in  nature,  co\  ering  many  areas 
of  support,  or  more  specific,  requiring  support  for  only  one  area. 

The  typical  contract  types  for  engineering  and  technical  ser\  ices  encountered  by 
this  researcher  were  I)  Cost  Plus  Fixed  Fee  (CPFF),  2)  Time  and  Materials  (T&M),  3) 
Indefinite  Delivery  Indefinite  Quantity  (IDIQ),  and  4)  Cost  Plus  Award  Fee  (CPAP^.  As 
an  observation,  activities  used  either  the  CPFF  and  CPAF  or  T&M  and  IDIQ.  Only  one  of 
the  activities  encountered  during  this  research  used  a  mixture  of  Cost  Pius  and  T&M  type 
contracts. 


F.  DESCRIPTION  OF  "BEST  VALUE"  SOURCE  SELECTION  FOR 

ENGINEERING  AND  TECHNICAL  SERVICES 

As  discussed  in  Section  A,  "best  value"  is  generally  defined  in  terms  of  a  source 
selection  evaluation  process  that  is  normally  reserxed  for  non-routine  ser\  ice  requirements. 
"Best  \  alue"  source  selection  employs  e\  aluation  factors  to  discriminate  betw  een  offerors. 
The  evaluation  can  be  either  formal  or  streamlined.  Both  techniques,  as  applied  to 
engineering  and  technical  services,  are  described  below  after  a  brief  discussion  of 
acquisition  planning. 

1 .  Acquisition  Pianning 

A  key  in  "best  value"  source  selection  is  acquisition  planning.  Acquisition 
planning  considers  elements  including  a  clear  definition  of  the  requirement,  objectives  of 
the  procurement,  and  establishes  the  composition  of  the  acquisition  team.  Each  element  is 
critical  to  the  success  of  the  procurement.  The  products  of  acquisition  planning  include  a 
well  defined  PWS  and  a  Source  Selection  Plan  (SSP).  Although  not  always  required,  and 
depending  on  the  designation  of  the  program  or  dollar  value  of  the  acquisition,  the 
advanced  planning  required  to  effectively  select  a  source  dictates  the  need  for  a  written 
acquisition  plan. 

A  typical  SSP  consists  of  two  parts.  Part  One  will  pro\'ide  an  overview  of  the 
acquisition  strategy  and  describe  the  organization,  membership,  and  responsibilities  of  the 
source  selection  team.  Part  Two  identifies  the  evaluation  criteria  and  the  detailed 
procedures  for  proposal  evaluation  [Ref.  12;p.  16].  It  is  of  utmost  importance  to 
thoroughly  plan  the  evaluation  criteria  to  be  used  in  evaluating  proposals.  Developing  the 
appropriate  evaluation  criteria  is  perhaps  the  most  important  aspect  of  acquisition  planning 
and  is  considered  to  be  the  most  difficult. 

Part  Two  of  the  SSP  also  contains  the  procedures  to  be  implemented  for  evaluating 
contractor  proposals.  The  topics  to  be  covered  will  depend  on  the  size  and  complexity  of 
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the  acquisition,  and  may  include;  1)  the  nature  and  scope  of  the  evaluation,  2)  preparation 
and  training  of  the  personnel  in\ol\cd,  3)  evaluation  committees,  4)  scoping  techniques, 
5)  risk  assessment,  6)  price  or  cost  evaluation,  and  7)  documenting  the  e\  aluation.  [Ref. 
12:Appendi\  B] 

Upon  completion  of  the  SSP,  it  is  necessary  to  solicit  proposals  from  industry. 
This  action  is  initiated  through  the  RFP.  All  sections  of  the  RFP  are  important  to  the 
successful  evaluation  of  proposals.  However,  Sections  B,  C,  L,  and  M  are  especially 
vital.  Sections  B  and  C,  "PWS/Specifications,"  contain  the  Government's  requirement.  It 
is  essential  to  ha\e  a  clear  and  concise  definition  of  the  requirement.  Section  L, 
"Instructions,  Conditions,  and  Notices  to  Offerors  or  Quoters,"  is  used  to  deliver 
instructions  to  offerors  on  how  to  prepare  and  deliver  this  proposal.  Section  M, 
"Evaluation  Factors  for  Award,"  must  contain  the  following  information  in  accordance  with 
FAR  subpart  15.605  (E); 

The  solicitation  shall  clearly  state  the  evaluation  factors,  including  cost  or  price  and 
any  significant  subfactors,  that  will  be  considered  in  making  the  source  selection  and 
their  relative  importance.  Numerical  weights,  which  may  be  employed  in  the  evaluation 
of  proposals,  need  not  be  disclosed  in  solicitations.  The  solicitation  shall  inform 
offerors  of  minimum  requirements  that  apply  to  particular  evaluation  factors  and 
significant  subfactors.  [Ref.  8;p.  15-20] 

Additional  steps  can  be  taken  to  ensure  the  RFP  is  as  complete  as  possible.  These 
steps  include  the  utilization  of  a  draft  RFP  or  presolicitation  conference  w  hich  allow 
industry  the  opportunity  to  carefully  review'  the  requirement  and  provide  feedback  on 
possible  problem  areas. 

2.  Formal  Source  Selection  Method 

The  establishment  of  a  specific  evaluation  and  decision  organization  is  what 
designates  certain  source  selections  as  formal.  The  formal  source  selection  method  is  a 
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time  consuming  and  administratively  expensive  process  generally  reserved  for  major 
weapon  systems  and  other  complex  requirements.  It  can  also  provide  the  greatest 
flexibility  to  DoD  in  selecting  the  most  advantageous  contractor  proposal. 

Responsibilities  of  the  formal  source  selection  organization  from  DLA  Handbook 
4105.3  are  listed  below: 

1 .  The  Source  Selection  Authority  (SSA)  is  the  individual  responsible  for  making  the 
final  decision  on  recommendations  from  the  Source  Selection  Advisory  Council 
(SSAC)  and  the  Source  Selection  Evaluation  Board  (SSEB).  The  SSA  is  normally 
a  high  ranking  official  in  the  organization  possessing  a  broad  perspective  of  the 
organization's  mission  and  is  in  the  best  position  to  make  a  decision  based  on  the 
agency  and  its  requirements. 

2.  The  SSAC  assists  the  SSA  and  SSEB  in  an  advisory  status.  It  reviews  and 
appro\  es  factors  and  their  weighting  used  in  the  source  selection  evaluation  and 
appoints  chairpersons  and  members  of  the  SSEB. 

3 .  The  SSEB  is  responsible  for  the  dexelopment  of  ex  aluation  standards,  evaluation  of 
proposals  in  accordance  with  those  standards  and  presentation  of  findings  to  the 
SSA  and  SSAC.  The  standards  relate  to  technical  and  cost  factors  for  each 
proposal.  The  purpose  of  the  SSEB  is  an  agreement  on  the  strengths  and 
weaknesses  of  each  proposal  against  the  SSP.  These  strengths  and  weaknesses  are 
presented  to  the  SSA  as  an  individual  proposals.  Proposals  are  not  compared  bv 
the  SSEB. 

4.  The  contracting  officer  is  responsible  for  preparing  the  acquisition  plan,  preparing 
the  solicitation,  evaluating  proposal  costs  to  determine  its  reasonableness  and 
making  the  aw  ard.  The  contracting  officer  is  normally  assigned  as  the  chairperson 
of  the  SSEB  cost  section  which  evaluates  proposal  costs  for  reasonableness  and 
realism.  [Ref.  3:p.  II-3] 

a.  Evaluation  Factors 

The  process  of  formal  source  selection  facilitates  an  agency's  ability  to  make 
the  most  advantageous  procurement  decision  through  the  use  of  evaluation  factors. 
Ev  aluation  factors  are  used  to  discriminate  between  offerors.  A  procuring  agency  has  great 
latitude  in  determining  which  factors  should  be  utilized.  Three  requirements  on  the 
selection  of  evaluation  factors  from  DLA  Handbook  4105.3  are: 

1 .  The  factor  must  be  variable,  i.e.,  there  must  be  a  reasonable  expectation  of  variance 
between  offerors. 
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2.  The  variance  musl  be  measurable.  This  does  not  mean  it  must  be  quantifiable; 
qualitative  measures  are  equally  valid. 

3.  The  factor  must  be  determinant.  The  Comptroller  General  has  stated  in  several 
decisions  that  use  of  an  area  as  an  evaluation  factor  is  valid  only  if  the  agency  's 
needs  warrant  a  comparative  evaluation  of  those  areas.  The  FAR  reinforces  this  b\ 
stating  that  the  evaluation  factors  will  include  only  those  factors  that  ha\  e  an  imjjact 
on  the  source  selection  decision.  The  simplest  w  ay  to  assess  determinance  is  to  ask 
vourself:  "is  the  Government  willing  to  pay  more  for  higher  merit  in  this  factor?" 
[Ref.  3;p.  II-5] 

Typical  factors  utilized  in  the  source  selection  process  for  engineering  and 
technical  services  include  technical,  management  and  cost.  In  accordance  with  FAR  part 
15.605(b),  cost  or  price  and  quality  are  mandator\-  evaluation  factors  in  all  procurements. 
Quality  is  normally  considered  in  the  context  of  all  factors  including  the  technical 
excellence,  management  capability  or  past  performance  of  an  offeror.  Cost  is  listed  in 
Section  M  as  a  separate  e\  aluation  factor. 

In  order  to  facilitate  the  evaluation  process,  factors  are  normally  broken  down 
into  subfactors.  An  example  of  a  technical  factor  broken  down  into  subfactors  from  an 
RFP  by  the  Naval  Regional  Contracting  Center  (NRCC),  San  Diego,  Long  Beach 
Detachment,  is  presented  below : 

Factor  Technic  al 

Subfactors:  a)  Technical  Approach 

b)  Corporate  Experience 

c)  Personnel  Qualifications 

d)  Management  Plan 

e)  Quality  Assurance  Plan 

Factors  and  subfactors  are  weighted  in  accordance  with  their  relative 
importance.  These  weights  are  communicated  to  offerors  via  the  solicitation.  This  can  be 
accomplished  by  listing  factors  and  significant  subfactors  in  their  relative  order  of 
importance,  or  by  using  adjective  descriptions  of  the  factor's  imjxirtance,  i.e.,  "Technical  is 
more  imjxjrtant  than  management.  Technical  and  management  factors  combined  are 
slightly  more  important  than  cost." 


17 


Factors  are  also  designated  as  variable  in  relati\  e  importance.  For  instance,  if 
the  difference  between  two  offerors  is  insignificant  on  the  most  important  factor,  the 
difference  in  lesser  factors  may  become  more  important  in  the  overall  evaluation. 

In  evaluating  factors  and  subfactors,  the  SSEB  utilizes  evaluation  criteria  such 
as  1)  the  soundness  of  the  technical  approach,  2)  compliance  w  ith  the  requirements,  3) 
excellence  of  designs,  and  4)  demonstrated  understanding  of  the  requirement  [Ref.  3:p. 
II-8].  Multiple  assessment  criteria  can  result  in  a  matrix  evaluation  arrangement  which  can 
assist  in  visualizing  the  o\  erall  proposal  e\  aluation. 
b.  Rating  Systems 

Rating  descriptions  of  each  factor  could  be  based  on  point  systems,  adjective 
grades,  colors,  or  a  combination,  depending  on  the  agency  and  acti\’ity  within  the  agency. 
For  example,  the  Air  Force  is  required  by  regulation  to  use  colors  as  a  rating  system, 
w'hereas  Navy  acti\  ities  contacted  by  this  researcher  haNe  used  points,  adjectives,  a 
composite  system  of  points  and  adjectix  es,  and  colors.  Each  system,  by  its  nature,  has 
advantages  and  disadvantages.  How  ever,  if  used  properly  as  a  tool  of  the  SSA,  the  results 
should  be  the  selection  of  the  "best  value"  offeror. 

(1)  Point  Systems.  Of  all  the  rating  systems  ax  ailable,  point  systems  are  the 
most  precise.  They  are  typically  based  on  a  100  or  1000  point  scale.  Points  are 
accumulated  during  proposal  ex  aluation  in  the  form  of  numerical  scores,  w  hich  allow  the 
SSA  to  make  total  point  comparisons  between  competing  proposals.  For  example,  one 
offeror  may  achieve  a  total  score  of  75  points  on  its  proposal  compared  to  a  superior 
proposal  which  achieved  a  score  of  85  points.  Point  systems  do  not  alloxv  the  flexibility  of 
adjective  ratings  and  colors  in  making  tradeoffs  between  evaluation  factors.  This  is  due  to 
the  precise  grades  given  to  proposals  xvhere  the  differences  are  sometimes  difficult  to 
equate  with  a  precise  numerical  score. 


18 


(2)  Composite  Rating  Systems.  Composite  rating  systems  consist  of 
numerical  scores  attached  to  an  adjecti\  a]  or  color  descriptor.  Composite  sy  stems  utilize  a 
numeric  scale  and  assign  an  adjecti\  al  or  color  descriptor  depending  upon  where  the  total 
points  achie\  ed  by  the  prop>osal  fall  on  the  scale.  For  example,  on  a  numeric  scale  of  0- 
100,  any  propxjsal  achie\  ing  a  numeric  score  between  90-100  total  points  will  be  assigned 
an  adjecti\a]  descriptor  of  "outstanding"  and  a  color  descriptor  of  "blue".  Composite  rating 
systems  eliminate  the  need  to  differentiate  when  proposals  are  only  two  or  three  points 
apart  under  a  point  system.  Ho\\e\  er,  there  may  be  a  considerable  difference  between 
proposals  on  the  high  emd  low  ends  of  the  adjectival  grade  of  "outstanding". 

(3)  Adjecti\  al  and  Color  Systems.  Employing  strictly  adjecti\  al  grades  or 
colors  as  a  rating  system  permits  the  SSA  the  most  flexibility  in  making  contract  awards 
because  it  does  not  constrain  the  SSA  to  making  awards  based  on  high  point  scores 
associated  with  point  and  composite  systems.  In  fact,  using  strictly  adjectives  or  colors 
requires  the  SSA  to  focus  more  on  concrete  differences  in  proposals  rather  than  relying  on 
point  scores.  A  disadvantage  of  adjectix  es  and  colors  is  the  lack  of  preciseness  sometimes 
desired  by  the  technical  codes  evaluating  an  offeror’s  proposal.  As  an  example  of  rating 
systems,  consider  Table  1 : 


Table  1 


Score 

Scale 

Color  Rating 

Offeror  A 

92 

90-100 

HHI 

Blue 

Offeror  B 

88 

80-89 

Excellent 

Green 

The  difference  between  Offeror  A  and  Offeror  B  equals  only  four  points  in 


numerical  scores,  yet  equates  to  an  adjectival  rating  difference  between  an  "excellent"  and 
"outstanding"  offeror.  If  the  numerical  grades  were  95  for  Offeror  A  and  85  for  Offeror  B, 
the  adjective  rating  is  more  descriptive. 
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The  Naval  Supply  Systems  Command  (NAVSUP)  prefers  usage  of  the 
composite  method  but  recognizes  that: 

A  variety  of  (rating  systems)  methodologies  have  been  advocated  and  successfully 
employed  throughout  the  Na\  y  Field  Contracting  System  (NFCS)  w  hich  offer  different 
ad\antages  and  drawbacks.  .  .acti\  ities  are  authorized  to  employ  the  system  w  hich 
appears  most  advantageous  to  their  needs.  [Ref.  2:  p.  11-12] 

DLA  also  recognizes  a  variety  of  rating  systems  and  stresses  systems 

which; 

.  .  .allow  the  SSA  the  maximum  flexibility  in  making  tradeoffs  between  evaluation 
factors.  Numerical  systems  present  the  most  problems  in  this  area.  [Ref.  3:p.  8-11] 

Unlike  NAVSUP  and  DLA,  the  Air  Force  mandates  the  use  of  a  color 
coded  rating  ssstem. 

Use  colors  to  indicate  proposal  ratings  at  the  factor  and  subfactor  le\'el.  At  the  area 
and  item  level,  color  codes  shall  also  be  used  to  depict  the  rating  and  will  be 
accompanied  by  a  consistent  narrative  assessment  detailing  the  basis  for  the  rating. 
[Ref.  6;p.  11] 

Regardless  of  the  rating  description  utilized  during  the  source  selection,  it 
is  the  responsibility  of  the  SSA  to  make  the  source  selection  decision  based  on  the  relati\  e 
strengths  and  weaknesses  of  each  proposal.  Rating  descriptions  are  a  tool  of  the  SSA. 
Two  proposals  e\  aluated  as  equal  overall  under  a  point  system,  adjecti\  es,  or  colors  may¬ 
be  considerably  different  upon  examining  the  discriminators  between  the  offerors.  If  two 
competing  proposals  are  evaluated  equal  overall,  one  of  the  key  points  is  to  look  back  at  the 
RFP  and  the  SSP  and  determine  which  factor  was  considered  most  important.  It  may 
happen  w  here  one  offeror  has  done  very'  well  on  factors  not  considered  the  most  important 
and  overall,  achieved  an  equal  grade  with  the  offeror  w  ho  did  well  on  the  factor  considered 
most  important.  This  requires  an  investigatiori  before  awarding  the  contract  and  is  why- 
narrative  comments  from  the  technical  e\-aluation  board  are  crucial. 
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The  bottom  line  in  designing  a  rating  system  is  to  maximize  useful 
information  to  the  SSA  without  requiring  an  extensi\e  re\  iev\  of  each  proposal  by  the  SSA. 
This  is  achiex  ed  through  a  proper  design  and  e\aluation  of  factors  b)  the  SSEB  and  a 
presentation  of  each  contractor's  strengths  and  weaknesses  to  the  SSA. 
c.  Cost  Realism  Analysis 

The  cost  realism  analysis  is  completed  by  the  cost  panel  of  the  SSEB  in 
conjunction  with  the  technical  panel.  It  is  used  in  the  evaluation  of  cost  reimbursement  type 
contracts  even  \\  hen  adequate  price  competition  exists.  A  cost  realism  analysis  is  generally 
not  used  w  hen  fixed  price  and  time  and  materials  type  contracts  are  contemplated  because 
prices  are  fixed  and  there  is  no  opportunity  for  an  upward  adjustment  in  the  cost  proposal. 
However,  like  all  contracting  actions,  fixed  piice  and  time  and  materials  type  contracts  do 
require  a  reasonableness  evaluation.  As  defined  in  the  DEARS,  Section  215.801,  the  cost 
realism  analysis  is; 

A  rcN’iew  of  the  cost  position  of  an  offeror's  proposal  to  determine: 

1 .  If  the  o\  erall  costs  proposed  are  realistic  for  the  u  ork  to  be  performed; 

2.  If  the  costs  reflect  an  offeror's  understanding  of  the  requirements  of  the 
proposed  contractor; 

3.  If  the  costs  are  consistent  w  ith  the  various  elements  of  the  technical  proposal. 
[Ref.  11:  p.  215-2] 

For  example,  if  an  offeror  neglects  certain  costs  required  to  complete  the  technical 
requirements,  it  may  indicate  an  incomplete  or  poor  understanding  of  those  technical 
requirements  and  an  increased  risk  to  the  Government.  A  cost  realism  analysis  is  also 
conducted  for  use  in  the  cost/technical  tradeoff  analysis  if  required  prior  to  contract  award. 
In  this  respect,  the  cost  realism  analysis  evaluates  the  contractor's  costs  to  determine  their 
reliability.  If  the  proposed  costs  are  found  to  be  inaccurate,  the  costs  are  adjusted.  For 
example,  if  an  offeror  proposes  unrealistically  low  labor  rates  in  a  Best  and  Final  Offer 
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(BAFO)  without  justification,  costs  would  be  adjusted  upward  to  reflect  accurate  costs  in 
accordance  with  the  cost  realism  assessment.  The  upward  adjusted  cost  is  used  in 
conjunction  with  the  technical  e\  aluation  in  the  overall  proposal  assessment. 
d.  Cost  Factor 

The  cost  factor  for  evaluation  is  weighted,  although  it  is  never  scored  in  "best 
value"  evaluations.  Weighting  costs  means  an  assignment  is  made  indicating  the  relati\  e 
importance  of  cost  in  comparison  to  the  technical  proposal.  For  instance,  technical  and 
management  factors  combined  may  be  slightly  more  important  than  cost.  If  translated  to 
points,  this  may  equate  to  a  60/40  split  which  means  that  out  of  100  total  points,  60  are 
attributed  to  technical  and  management  factors,  while  40  are  accounted  for  in  cost.  Only  a 
few  NAVSUP  acti\  ities  w  ere  observed  during  this  research  to  use  a  point  breakdown  of 
cost  to  technical  factors.  All  other  Navy  activities,  the  Air  Force,  Army  and  NASA  claimed 
to  evaluate  the  cost  factor  by  assigning  a  relative,  not  a  numerically  specific  breakdown. 
Evaluating  cost  against  other  factors  is  thereby  more  flexible  and  less  dependent  on  a 
mathematical  formulation. 

The  scoring  of  the  cost  factor,  which  is  not  permitted  by  any  DoD  agency  or 
NASA,  is  the  assignment  of  points,  adjective  grades,  or  color  ratings  to  the  accuracy  of 
costs  in  the  proposal.  The  reason  this  practice  is  prohibited  is  because  scoring  can  skew  an 
ev'aluation  toward  a  contractor  with  accurate  costs  but  a  marginal  technical  proposal.  Even 
more  significant,  is  the  possibility  of  making  a  technically  unacceptable  proposal  acceptable 
for  award  based  on  the  accuracy  of  costs  alone.  For  example,  if  an  offeror's  costs  are 
determined  to  be  accurate  through  a  cost  realism  analysis,  that  offeror  would  receive 
additional  points  even  though  the  technical  proposal  was  marginal  or  unacceptable.  The 
additional  points  for  cost  accuracy  may  make  a  marginal  or  unacceptable  proposal  eligible 
for  contract  award. 


e.  Competitive  Range  Determination 


Upon  completion  of  the  initial  evaluation  phase  and  before  discussions,  a 
competitive  range  determination  is  made  [Ref.  3:p.  11-16].  This  determination  is  made  by 
the  contracting  officer  with  input  by  the  technical  panel.  The  determination  is  particularly 
important  in  a  formal  source  selection  u  here  costs  associated  with  proposal  preparation  and 
e\'al  nation  are  great. 

In  determining  the  competitive  range,  a  tradeoff  between  a  broad  competitive 
base  and  the  opportunity  of  award  requires  consideration.  In  such  a  case,  it  is  generally 
considered  prudent  to  err  in  fa\’or  of  a  broader  range.  FAR  subpart  15.609(a)  states  that 
"when  there  is  doubt  as  to  whether  a  proposal  is  in  the  competiti\  e  range,  the  proposal 
should  be  included."  [Ref.  8:p.  15-22] 

Competiti\e  range  determinations  are  made  based  on  all  factors  in  the 
solicitation  and  in  the  same  manner  a  proposal  is  e\’aluated  for  contract  award.  Competi:i\'e 
ranges  are  established  on  the  basis  of  comparative  assessments  of  proposals  in  which  an 
average  proposal  may  be  rejected  based  upon  the  outstanding  merits  of  others.  As 
previously  noted,  contracting  officers  must  be  careful  not  to  eliminate  small  businesses 
based  on  responsibility  factors  when  a  certificate  of  competency  determination  is  required. 
Instead,  a  comparative  assessment  based  on  the  merits  of  other  proposals  should  be 
employed  when  eliminating  any  proposal  from  the  competitive  range. 

/.  Discussions 

The  next  stage  of  formal  source  selection  is  called  discussions.  Discussions 
are  defined  in  the  FAR  part  15.601  to  include; 

. .  .any  written  or  oral  communication  between  the  Go\’emment  and  the  offeror  that 
involves  information  essential  for  determination  of  acceptability  of  a  proposal  or  that 
provide  the  offeror  to  revise  or  modify  its  proposal.  [Ref.  8:p.  15-19] 


Apparent  clerical  mistakes  may  be  corrected  prior  to  the  establishment  of  a  competitix  e 
range  and  without  entering  into  discussions  where  they  qualify  as  clarifications. 


There  is  a  fine  line  between  clarification  and  discussions,  and  as  a  result,  all 
correspondence  between  the  evaluation  board  and  the  offeror  should  be  first  directed 
through  the  contracting  officer  [Ref.  3:p.  11-18].  If  responses  to  clarification  requests 
represent  modifications  or  revisions  to  an  offeror’s  proposal,  those  modifications  or 
revisions  should  not  be  forwarded  to  the  evaluation  board  since  it  may  change  the 
evaluation  in  favor  of  the  offeror  making  the  change.  The  changes  could  possibly  add  an 
offeror  to  the  competiti\’e  range  who  was  previously  not  considered  susceptible  for  award. 
Clarifications  for  offerors  already  in  the  competitive  range  can  be  deferred  until  discussions 
ha\’e  commenced. 

Discussions  are  conducted  by  the  contracting  officer  with  technical  evaluators 
present  as  requested.  The  idea  is  to  clear  up  questions  regarding  the  offeror's  proposal  and 
inform  the  offeror  of  deficient  areas  prior  to  BAFOs.  Extreme  care  must  be  taken  during 
discussions  so  as  not  to  "technically  level"  the  offerors  by  providing  solutions  to 
deficiencies  in  the  proposals  noted  during  the  evaluation  [Ref.  3:p.  11-18].  Another  practice 
to  avoid  is  that  of  technical  transfusion,  where  one  offeror's  superior  technical  solution  is 
communicated  to  lower  rated  technical  offerors  [Ref.  3:p.  11-18].  The  point  is  to  avoid 
discussing  solutions  to  the  deficiencies,  and  instead  communicate  why  the  evaluation  team 
determines  certain  areas  of  the  proposal  as  deficient. 

Cost  discussions  permit  the  contracting  officer  to  question  offerors  on  the  cost 
elements  of  their  proposals.  For  instance,  a  cost  proposal  may  lack  certain  cost  elements, 
or  the  cost  proposed  may  be  unrealistic  in  view'  of  the  level  of  effort  expected.  Both 
deficiencies  may  be  addressed  during  discussions.  This  allows  the  offeror  the  opportunity 
to  reevaluate  its  position  for  possible  revision  in  the  BAFO  [Ref.  3:p.  11-18]. 
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Agencies  were  recently  empowered  with  the  opportunitx'  to  make  awards 
without  discussions  to  other  than  the  low  price  offeror  prox  ided  the  cost  is  competitix  e  and 
the  competing  offeror  is  not  susceptible  for  award.  The  revision  is  a  result  of  Section  802 
of  the  1991  National  Defense  Authorization  Act  and  incorporated  in  FAR  15.610  [Ref.  8:p. 
15-22].  Past  policy  required  axvards  xvithout  discussions  be  made  to  the  offeror  with  the 
loxx  price  bid.  Axvarding  xx  ithout  discussions  saves  the  contracting  agency  and  offerors 
time  and  resources  associated  with  conducting  discussions  and  subsequent  BAFOs. 
g.  Best  and  Final  Offers  (BAFO) 

BAFOs  prox  ide  all  offerors  in  the  competitive  range  xx  ith  a  common  cutoff  date 
to  submit  modifications  to  their  proposals,  ideally  directed  at  the  deficiencies  noted  during 
discussions.  There  is  no  limit  on  the  changes  an  offeror  can  make  during  the  BAFO  stage 
although  it  is  possible  to  require  offerors  to  illuminate  changes  for  ease  of  evaluation. 
[Ref.3:p.  11-19] 

After  receipt  of  BAFOs,  all  offers  in  the  competitive  range  are  evaluated  once 
more  in  accordance  xxith  the  factors  and  evaluation  criteria  listed  in  the  SSP.  The  only 
difference  is  noxx'  the  evaluation  should  focus  on  deficiencies  noted  during  discussions  and 
addressed  in  the  BAFOs.  At  completion,  the  strengths  and  xveaknesses  of  each  proposal 
should  be  presented  to  the  SSA  xvho  makes  the  final  decision  that  is  documented  in  the 
Source  Selection  Decision  Memorandum  (SSDM).  The  SSDM  should  provide  the 
decision,  supporting  rationale  and  direction  to  the  contracting  officer  to  make  an  award. 
[Ref.  3:p.  11-19] 

2.  Streamlined  Source  Selection  Method 

The  streamlined  source  selection  method  encompasses  the  same  procedures  of  the 
formal  source  selection  method  and  permits  the  procuring  agency  the  flexibility  to  award 
contracts  based  on  factors  other  than  price  alone.  The  streamlined  source  selection  method 
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is  normally  reserved  for  lower  cost,  less  complex  procurements.  It  is  also  more  commonh' 
used.  In  accordance  with  the  DLA  Handbook  4105.3,  the  streamlining  can  be 
accomplished  in  four  w  a\  s; 

1 .  Streamlining  the  source  selection  organization  by  limiting  the  number  of  persons 
involved  in  the  source  selection  process;  i.e.,  limiting  membership  on  the  SSEB. 

2.  Limiting  the  number  of  factors  and  subfactors  used  in  the  evaluation  process. 

3 .  Limiting  the  length  of  technical  proposals  to  a  certain  number  of  pages. 

4.  Standardizing  the  evaluation  factors  and  criteria  to  reduce  development  time  and 
amount  of  explanation  and  justification  required  in  the  evaluation.  |kef.  3:p.  III-l] 

All  agencies  within  DoD  and  NASA  have  these  streamlined  procedures  w'hich  are 
sometimes  referred  to  as  the  informal  source  selection  method.  Minimum  dollar  amounts 
for  use  of  the  streamlined  source  selection  method  may  vary  among  agencies.  The  general 
guidance  revolves  around  the  cost  of  the  process  compared  to  the  benefit  to  be  gained  b^ 
employment  of  the  process.  For  instance,  an  activity  may  not  feel  that  the  support  they  are 
getting  from  a  previously  awarded  low  price  technically  acceptable  offeror  is  adequate. 
Prior  to  recompeting  the  sen  ice,  the  activity  decides  through  a  cost  benefit  analysis  that  the 
streamlined  source  selection  method  will  be  used  because  it  will  allow  the  activitj-  to  obtain 
better  services  by  using  evaluation  factors  other  than  price  alone. 

The  streamlined  source  selection  method  is  related  to  initiatives  currently  being 
pursued  at  NASA  and  the  Defense  Nuclear  Agency  (DNA)  to  reduce  the  Procurement 
Administrative  Lead  Time  (PALT).  The  Fast  Track  Procurement  program,  instituted  by 
DNA,  strives  to  reduce  PALT  to  120  days  on  service  procurements  less  than  $2  million 
[Ref.  13].  It  does  so  by  reducing  the  number  of  evaluation  factors,  length  of  technical 
proposals  and  incorporating  a  structured  process  with  recommended  milestones. 
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As  expected,  streamlining  one  aspx^ct  of  the  process  w  ill  carr)’  o\  er  into  another. 
For  example,  reducing  the  length  of  the  technical  proposal  reduces  the  number  of 
evaluation  board  members  and  the  time  required  to  conduct  the  evaluation. 

The  streamlined  organization  usually  consists  of  the  procurement  official,  or 
contracting  officer  who  acts  as  the  SSA  and  the  SSEB.  The  SSAC  is  either  eliminated  or 
integrated  into  the  SSEB.  [Ref.  3:p.  II1-2] 

G.  SUMMARY 

Chapter  II  provided  a  definition  of  the  "best  value"  concept  to  be  utilized  in  this  stud\ 
and  discussed  when  it  is  appropriate  to  employ  "best  value"  evaluations.  Chapter  II  also 
provided  a  definition  and  discussion  of  engineering  and  technical  ser\  ices.  The  chapter 
was  concluded  w  ith  a  generic  discussion  of  acquisition  planning  and  the  "best  value" 
source  selection  methods  for  engineering  and  technical  sen  ices. 

Chapter  III  w  ill  provide  data  from  a  suney  questionnaire  used  to  conduct  telephone 
and  personal  inten  iews.  The  sun  ey  questionnaire  solicited  opinions  from  Government  and 
industry  personnel  on  the  subject  of  "best  value"  and  its  use  in  the  procurement  of 
engineering  and  technical  sen  ices. 


III.  "BEST  VALUE"  SURVEY  QUESTIONNAIRE 


A.  INTRODUCTION 

Chapter  III  is  a  compendium  of  responses  resulting  from  telephone  and  personal 
interviews  collected  through  the  use  of  a  survey  questionnaire.  The  questionnaire  was 
developed  based  upon  an  analysis  of  the  current  literature  and  discussions  with 
procurement  personnel  who  utilize  "best  value"  techniques  to  procure  engineering  and 
technical  ser\  ices.  The  questions  were  left  open-ended  with  the  idea  of  eliciting  responses 
which  would  correlate  into  meaningful  issues  for  further  discussion.  Respondents  to  the 
surN'ey  were  randomly  selected,  and  include  procurement  and  technical  personnel  from  both 
Government  and  industry.  Prior  to  the  commencement  of  an  inter\'iew,  the  inten'iewee 
was  informed  of  the  nature  of  the  study  and  advised  that  comments  received  by  the 
researcher  will  be  kept  on  a  non-attribution  basis.  Inter\’iewees  are  listed  in  Appendix  B. 

In  addition  to  the  data  collected  during  interviews,  information  was  collected  from 
local  instructions  and  training  aids,  three  source  selection  plans,  nine  requests  for  proposal, 
and  five  source  selection  decision  memorandums  representing  17  different  contractual 
actions  from  eight  separate  activities.  The  Federal  Legal  Information  Through  Electronics 
(FLITE)  database  was  screened  for  Board  of  Contract  Appeals  (BCA)  and  Comptroller 
General  (GAO)  decisions  for  the  key  terms  "best  value",  "greatest  value"  and  "best  buy" 
with  regard  to  sen  ice  contracts  awarded  since  1988. 

The  data  presented  are  not  statistically  significant  nor  do  they  apply  to  all  DoD 
activities  but  represent  a  significant  sample  of  the  issues  and  problems  experienced  by  the 
interviewees.  The  format  for  presentation  of  the  questionnaire  commences  with  the 
question,  a  discussion  of  the  overall  responses,  and  in  several  cases,  a  representative 


sampling  of  responses.  Separate  sur\  ey  questionnaires  were  utilized  for  Go\  emment  and 
contractor  personnel.  Questions  are  presented  in  Appendix  C.  Major  issues  that  were 
identified  during  the  inten  iew  s  are  defined  and  analyzed  in  Chapter  IV. 

B.  GOVERNMENT  QUESTIONNAIRE 
1 .  Question  #1 

How  would  you  define  "best  (greatest)  value"? 
a.  Discussion  ^ 

Respondents'  definitions  of  the  term  "best  value"  were  consistent  with 
published  definitions  stating  that  "best  value"  equates  to  making  the  most  advantageous 
procurement  decision  based  on  factors  other  than  price  alone.  The  majority  defined  "best 
value"  in  terms  of  a  process,  alluding  to  a  method  of  source  selection  where  the 
Government  is  afforded  the  opportunity  to  exercise  judgment  in  considering  all  factors 
prior  to  awarding  a  contract.  AdjectiNes  including  "subjective,"  "reasoned,"  and  "sound 
business"  were  frequently  used  as  descriptors  for  the  w  ord  judgment  w  hen  defining  "best 
value".  Outside  the  Navy,  almost  50  percent  of  the  interviewees  were  not  familiar  with  the 
term  "best  value"  although  they  were  familiar  with  the  process  of  considering  all  factors 
including  price  during  proposal  e\  aluaiion.  Four  of  the  inten  iewees  outside  the  Navy, 
believed  that  "best  value"  was  a  Na\  y  expression  used  to  describe  source  selection  in 
which  the  Navy  used  a  mathematical  formulation  combining  the  scoring  of  costs  and 
technical  factors  in  making  a  procurement  decision.  The  Na\  y  was  the  last  DoD  agency  to 
eliminate  the  practice  of  numerically  scoring  costs.  The  practice  was  eliminated  several 
years  ago. 

Three  interviewees  offered  an  interesting  caveat  that  they  claimed  was  a 
common  misconception  among  procurement  and  non-procurement  personnel.  "Best  value" 
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in  their  opinions,  does  not  mean  procuring  more  than  the  minimum  requirements.  Instead, 
it  is  a  method  for  assessing  risk  and  in  turn  choosing  the  offeror  with  the  greatest 
probability  for  successful  contract  {jerformancc. 

A  final  ca\’eat  presented  by  a  high  ranking  Na\’y  procurement  official  considers 
"best  value"  as  an  objective  of  the  procurement  process  no  matter  what  the  product, 
seiA  ice,  or  method  of  procurement  utilized.  While  the  majority  of  viewpoints  tend  to 
equate  "best  value"  to  offering  a  premium  for  a  higher  quality  product  or  service,  this 
viewpoint  considers  "best  value"  as  an  objective.  At  times,  attaining  the  objective  requires 
the  payment  of  a  premium.  At  other  times,  "best  value"  may  mean  competing  a 
requirement  based  on  low  price  or  low  overall  cost.  This  decision  is  the  essence  of  "best 
value". 


b.  Responses 

Comments  reflecti\  e  of  responses  recei\  ed  during  inter\  ieu  s  are  listed  below: 

"Best  value"  is  a  conscious  departure  from  technically  acceptable/low  offeror 
procurements.  It  is  dependent  on  risk.  How  much  premium  are  we  (DoD)  willing  to 
pay  to  avoid  risk. 

"Best  value"  means  getting  the  most  in  terms  of  performance,  quality,  and 
suitability,  not  necessarily  related  to  price.  From  a  contract  administration  pxDint  of 
view,  it  is  money  put  in  up  front  compared  to  money  put  in  over  the  long  term. 

Under  "best  value",  technical  means  more  than  cost.  Cost  becomes  more  important 
as  technical  proposals  get  closer. 

"Best  value"  means  different  things  to  different  pjeople.  The  Government  reser\'es 
the  right  to  award  to  other  than  the  lowest  bidder  based  on  cost  and  other  factors. 

The  "best  value"  concept  involves  looking  at  the  entire  spectrum  of  factors  in 
making  an  award.  It  is  based  on  subjective  choices  between  offeror's  proposals. 

To  some,  "best  value"  means  achieving  the  most  bang  for  the  buck.  To  others,  it 
means  making  a  subjective  decision  on  projX)sals  which  the  activity  feels  offers  the 
most  ad\'antageous  procurement  decision  to  the  Government. 
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"Best  value"  is  not  choosing  the  low  bidder.  Nor  is  it  choosing  the  low  price 
technically  acceptable  offeror.  "Best  \aluc"  is  not  a  mathematical  conclusion.  "Best 
value"  is  a  reasoned  judgment  call  based  on  the  reasons  of  technical  ability, 
management,  and  cost  with  no  excuses  and  no  apologies. 

2 .  Question  #2 

Does  your  acti\  ity  use  numerical  scoring,  adjective  grades  or  colors  to  evaluate 
proposals  to  achieve  "best  value?"  Why  is  that  rating  system  used? 
a.  Discussion 

The  three  rating  systems  encountered  during  this  study  w'cre:  (1)  numerical 
scoring,  (2)  composite  methods  in  which  numerical  scores  are  attached  to  adjecti\  e  grades 
or  colors,  (3)  and  strictly  adjectival  descriptors  or  colors  without  any  numerical  points  or 
scales. 

Prop<'in''nis  of  utilizing  strictly  numerical  scoring  methods  feel  that  rating 
proposals  by  .lumerical  scores  is  the  most  direct  and  simplified  method  of  evaluation.  They 
also  claim  that  their  customers,  the  requiring  activities,  favor  numerical  scoring  as  a  method 
of  differentiating  between  offerors,  i.e.,  there  may  be  a  big  difference  between  a  technical 
score  of  81  and  86,  both  of  which  may  compute  to  an  "excellent"  adjectival  description. 

Composite  rating  systems  employ  both  numerical  scoring  methods  and 
adjectival  or  color  descriptors.  Acti\  ities  using  composite  rating  systems  appreciate  the 
ease  of  point  scoring  for  the  technical  evaluation  committee  and  the  flexibility'  of  adjectival 
or  color  descriptors  offered  to  the  SSA  in  making  award  decisions.  For  example,  a 
technical  proposal  achieving  a  numerical  score  betw'een  80-90  points  may  be  assigned  a 
grade  of  "excellent"  w  hile  a  technical  proposal  achieving  a  grade  between  91-100  w'ill  be 
assigned  a  grade  of  "outstanding".  A  composite  system  is  the  middle  ground  between  the 
other  two  systems,  employing  both  point  scoring  and  adjectival  or  color  descriptors. 

Activities  utilizing  strictly  adjectival  ratings  or  colors  stress  the  flexibility 
inherent  in  a  system  unconstrained  by  numbers.  Informally,  it  appears  that  activities  are 
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leaning  toward  the  more  flexible  rating  systems.  Two  Na\  y  acti\  ities  encountered  during 
this  study  recently  switched  from  numerical  rating  systems  to  strictly  adjecti\'al  systems. 
The  following  is  a  quote  taken  from  an  internal  memorandum  from  an  activity  currently 
utilizing  numerical  scoring  although  they  are  considering  changing  to  another  method.  The 
memo  is  dated  29  January'  1991. 

The  assignment  of  numerical  weights  too  often  results  in  the  requiring  activity  using 
it  as  a  crutch  rather  than  biting  the  bullet  and  coming  up  with  a  strong  commitment  as  to 
whether  a  higher  technical  score  really  offsets  a  higher  cost  or  in  fact,  did  not  offset  a 
higher  cost. 

Whate\’er  the  rating  system  employed,  all  activities  did  stress  that  the  rating 
system  is  utilized  as  a  tool.  The  narrative  from  the  technical  evaluation  committee 
supporting  the  rating  is  the  key  in  making  the  award  decision.  On  several  occasions,  the 
interviewees  inferred  that  the  head  of  the  procurement  acti\’ity  designated  the  rating  system 
as  a  matter  of  preference. 
b.  Responses 

We  use  numerical  scoring  for  technical  elements  only  based  on  a  1000  point  scale. 

At  one  point  or  another,  this  activity  has  used  all  three  (rating  systems),  although 
we  recently  stopped  using  colors  since  NAVSUP  didn't  like  the  way  we  attached 
meanings  to  their  usage.  At  present,  we  are  using  broad  adjectival  classes 
supplemented  by  point  scoring. 

We're  using  points  with  adjectives.  For  instance,  a  score  of  85-100  might  equate  to 
a  rating  of  "outstanding."  The  system  gives  us  flexibility  through  comparison  of 
adjectives  rather  than  total  point  scores. 

We  use  points  exclusively  and  typically  on  a  100  point  scale. 

Colors,  because  they  graphically  represent  the  overall  picture.  You  do  not  have  to 
go  into  great  depth  to  understand  colors.  Colors  are  more  flexible,  offering  a  wider 
latitude  to  the  SSA.  Besides,  the  Air  Force  has  a  regulation  banning  a  mathematical 
formula. 

Point  system  with  narrative. 

A  color  code  system  is  used  for  a  visual  aid  as  a  background  for  which  narrative  is 
most  important. 
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Adjecti\es.  Absolutely  no  numbers  are  used  during  the  e\'aluation  process.  We 
used  numbers  in  the  evaluation  process,  but  switched  to  adjecii\'es  about  tw  o  years 
ago.  At  first  I  was  doubtful,  but  now  I  would  not  use  any  other  rating  system. 

It  is  reaJlj’  academic  what  you  use;  numbers,  colors,  or  adjecti\  es.  It  all  depends  on 
the  SSA  and  what  he  or  she  feels  comfortable  with. 

3 .  Question  #3 

How  do  you  decide  a  certain  requirement  requires  the  use  of  the  "best  value" 
method? 


a.  Discussion 

Responses  to  this  question  were  consistent  with  published  guidance  noted  in 
Chapter  II.  Non-routine  ser\'ice  requirements  and  vague  PWSs  were  normally  cited  as 
indicators  necessitating  the  employment  of  "best  value"  evaluations.  Many  inter\  iewees 
responded  that  they  utilize  "best  value"  evaluations  almost  exclusi\  ely.  Others  utilize  "best 
value"  evaluations  less  frequentlj-.  Se\eral  respondents  stated  that  one  must  review  the 
type  of  contract.  Generally  speaking,  cost  type  contracts  should  result  from  the  use  of 
"best  value"  source  selection  procedures.  Augmenting  the  above  indicators  on  when  to 
utilize  "best  value"  evaluations  is  the  input  received  from  requiring  acti\'ities  and  contract 
review  boards  which  provide  guidance  on  the  type  of  procurement  necessarv-  for  a  specific 
requirement. 

b.  Responses 

It  depends  on  what  you  are  contracting  for.  The  contracting  officer  usually  gets  a 
good  idea  of  direction  from  the  branch  manager  and  others  during  contract  review 
panels.  The  contracting  officer  is  very  sensitive  to  this  recommendation. 

This  decision  requires  coordination  and  a  good  deal  of  definition.  History  has 
shown  that  engineering  and  technical  services  procured  on  a  low  offeror  basis  has  led 
to  miserable  support  overall. 

The  determination  is  made  in  conjunction  with  the  technical  code  requiring  the 
support. 

This  acti\  ity  utilizes  an  acquisition  strategy  board  prior  to  release  of  the  SOW  and 
RFP.  The  program  manager  makes  the  initial  call,  but  it  all  depends  on  the  risk 
assessment. 
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It  depends  on  the  technical  approach.  With  basic  senices,  a  responsibility 
determination  is  adequate.  For  more  complex  omnibus  or  engineering  support 
sen  ices,  "best  value"  in^■ol\'es  looking  toward  other  factors.  Contracts  and  technical 
have  had  arguments  about  this.  Just  because  a  requirement  calls  for  aerospace 
technology  doesn't  mean  a  run  of  the  mill  machine  shop  can't  perform  the  w  ork. 

It  depends  upon  the  requirements.  Basic  maintenance  sen  ices  do  not  usually 
require  "best  value".  However,  more  complex  services  do.  Preplanning  with  the 
technical  activity  is  verj’  important. 

Use  of  "best  value"  is  dependent  on  the  SOW  and  complexity  of  the  requirement. 

Depxjnds  on  how  important  the  technical  factors  are  to  the  job  requirement. 

Consultation  with  the  technical  activity  requiring  the  sen  ice.  You  also  need  to  take 
into  account  the  ty'pe  of  sen  ice  and  contract  type.  With  a  cost  reimbursement  contract, 
you  will  generally  lean  toward  "best  value." 

The  determination  relates  to  the  ser\  ice.  If  seio  ice  is  not  clearly  measurable  in  the 
area  of  acceptability,  one  will  generally  lean  toward  "best  value." 

4 .  Question  #4 

What  are  the  factors  and  subfactors  your  activity  uses  for  "best  value"  evaluations 
for  engineering  and  technical  serx  ices  contracts? 
a.  Discussion 

Responses  to  Question  #4,  during  telephone  interx  iew  s,  consistently  yielded 
the  factors  Technical,  Management,  and  Cost  as  a  basis  for  "best  \alue"  evaluations  for 
engineering  and  technical  sen  ice  contracts.  An  examination  of  a  database  of  local 
instructions,  three  source  selection  plans  (SSPs),  nine  requests  for  proposal  (RFPs)  and 
five  source  selection  decision  memorandums  (SSDMs)  from  17  unique  contracting  actions 
at  eight  different  activities  is  presented  in  Table  2  belou .  The  data  should  be  viewed  as  a 
rough  order  of  magnitude  tally  only  since  the  researcher  arbitrarily  assigned  subfactors  with 
reasonably  close  descriptions  to  other  subfactors.  For  instance,  the  subfactors  "manpower 
utilization  matrix"  and  "retention  policies"  were  included  with  the  subfactor  "management 
plan/approach." 
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FACTORS  AND  SUBFACTORS  UTILIZED  FOR  "BEST  VALUE" 
PROCUREMENTS  OF  ENGINEERING  AND  TECHNICAL  SERVICES 


TABLE  2 


FACTORS 

TIMES  LISTED 

Technical 

17 

Management 

5 

Cost 

17 

Corporate  Experience 

1 

SUBFACTORS 

TIMES  LISTED 

Corporate/Company 

Experience/Background 

14 

Technical  Approach 

14 

Personnel  Qualifications/Experience 

17 

Management  Plan/Approach 

12 

Sample  Tasks 

9 

Corporate  Resources/FacilUies 

6 

Technical,  Management,  and  Cost  are  the  key  factors  in  "best  value" 
procurements  for  engineering  and  technical  services  although,  in  this  sample,  Management 
is  generally  listed  as  a  subfactor  of  Technical.  Other  key  subfactors  are:  1)  Corporate 
Experience/Background,  \\hich  can  be  \  ie\\ed  as  the  le\  el  of  experience  in  identical  or 
similar  contracts  (these  data  are  used  by  several  activities  to  perform  a  qualitative  past 
performan  ^  assessment  on  the  offeror);  2)  Technical  Approach/  Technical  Understanding, 
which  equates  to  an  understanding  of  PWS  requirements;  and  3)  Personnel  Qualifications 
of  key  personnel  represented  by  resumes  supporting  minimum  requirements  for  experience 
and  education  of  proposed  key  personnel.  The  subfactor  Corporate  Resources/Facilities 
evaluates  the  offeror's  facilities  and  equipment,  and  Sample  Tasks  allow  the  offeror  to 
propose  solutions  to  tasks  which  have  been  previously  assigned  or  may  be  assigned  in  the 
future.  In  accordance  with  the  FAR,  Cost/Price  is  a  required  evaluation  factor  for  all 
proposals  [Ref.  8:p.  15-19]. 
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5 .  Question  #5 

How  are  past  performance  data  collected  and  utilized  in  "best  value"  source 
selection  for  engineering  and  technical  services  contracts? 
a.  Discussion 

According  to  inter\  ie\vees,  past  performance  is  a  factor  or  subfactor  in  the 
overwhelming  majority  of  "best  value"  evaluations  for  engineering  and  technical  services 
contracts.  The  process  of  collecting  past  performance  data  commences  with  the 
instructions  to  offerors.  Section  L,  of  the  RFP.  Section  L  requires  offerors  to  submit  a 
detailed  listing  of  previous  contracts  or  experiences  similar  to  the  current  PWS.  This 
requirement  is  normally  listed  under  the  evaluation  factor  "Corporate  Experience/ 
Background".  Once  the  data  on  experience  are  collected,  the  activity  utilizes  these  data  to 
research  an  offeror's  past  performance.  Past  performance  data  are  collected  through  the 
use  of  questionnaires  or  telephone  queries  to  the  procuring  activity  or  the  contract 
administration  office  having  cognizance  over  previous  efforts.  Past  performance  data  are 
typically  related  to  the  contractor's  adherence  to  proposed  cost,  schedule  and  general 
cooperativeness.  An  example  of  a  questionnaire  utilized  by  NAVAIR  to  collect  past 
performance  data  is  presented  in  Appendix  D.  The  process  of  past  performance  data 
collection  can  be  either  formal,  under  a  structured  program,  or  informal  where  one  :  .  tivity 
contacts  another  to  check  the  validity  of  data  provided  by  the  offeror  and  for  general 
information  on  a  contractor's  performance.  Both  methods  were  encountered  during  this 
research. 

Unlike  past  performance  data  for  tangible  items,  there  is  currently  no  rating 
system  like  Red/Yellow/Green,  Blue  Ribbon  Vendor,  or  the  Contractor  Performance 
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Rating  System  for  engineering  and  technical  serv  ices  contractors.  This  fact  is  related  to  the 
inherent  difficulty  of  assessing  quality  and  delivery  schedule  compliances  of  service 
contractors  performing  under  v  ague  PWSs. 

The  Naval  Sea  Systems  Command  (NAVSEA),  Washington,  D.C.  recently 
implemented  a  pilot  program,  the  Contractor  Performance  Index  (CPI),  to  fuinu  the  need  of 
providing  assessments  on  the  past  performance  of  engineering  and  technical  services 
contractors.  The  program  utilizes  supportable  and  verifiable  data  from  Contracting 
Officer's  Technical  Representatives  (COTR)  to  assess  the  positive  and  negative  aspects  of  a 
contractor's  past  performance  on  a  given  contract  over  a  specified  period  of  time.  CPI 
allows  for,  "Subjective  assessments  concerning  the  causes  or  ramifications  of  the 
contractor's  performance  (which)  should  be  provided;  however,  speculation  and  conjecture 
should  not  be  included."  [Ref.  14:pp.  1-2] 

One  of  the  major  issues  noted  during  interviews  germane  to  the  use  of  past 
performance  data  was  the  concern  of  past  performance  data  being  substituted  for  a 
responsibility  determination.  In  fact,  over  50  percent  of  the  interviewees  presented  w'ith 
Question  #5  said  that  care  must  be  taken  not  to  utilize  past  performance  data  in  place  of  a 
responsibility  determination.  What  this  means,  is  that  if  a  contractor  is  found  responsible 
to  prov  ide  the  services  desired  and  has  a  certificate  of  competency  from  the  Small  Business 
Administration,  the  Government  cannot  subsequently  utilize  data  based  solely  on  past 
performance  to  override  the  responsibility  determination  during  the  source  selection 
evaluation  process.  Two  interviewees  responded  that  they  believe  responsible  small 
businesses  with  certificates  of  competency  indicating  responsibility  were  eliminated  under 
the  guises  of  "best  value"  source  selections  based  on  past  performance.  This  w'as 
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evidenced  in  a  Comptroller  General  decision  in  the  matter  of  FLIGHT  International  Inc., 
discussed  in  Chapter  II  and  Comptroller  General  decision  B-231607  of  20  September  1988 
in  the  matter  of  Sanford  and  Son  [Ref.  3:p.  11-13]. 

All  activities  encountered  during  research  attribute  only  a  small  portion  of  the 
evaluation's  total  w  eight  to  past  performance.  In  reference  to  the  sample  of  contracting 
actions  collected  during  this  research  and  discussed  in  Question  #4,  it  was  found  that  the 
weightings  of  the  subfactor  Corporate  Experience/Background  ranged  between  2.5  and  15 
percent  of  the  total  evaluation  with  the  majority  of  the  weightings  ranging  between  5  and  10 
percent  of  the  total  evaluation.  As  a  result  of  this  relatively  small  w'eighting,  the  past 
performance  subfactor  should  not  overwhelm  other  factors  during  evaluation. 

A  recent  trend  in  past  performance  evaluations  discovered  during  personal 
interviews,  is  the  recognition  that  all  past  performance  data  provided  will  not  be 
complimentary  toward  the  offeror.  In  other  w-ords,  the  Government  is  urging  contractors 
to  present  all  past  performance  data  good  or  bad.  In  cases  where  poor  past  performance 
data  exist,  offerors  are  encouraged  to  provide  data  advising  the  procuring  activity  how  the 
problems  were  addressed,  corrected  and  the  current  status.  If  the  corrective  action  has 
merit,  the  offeror  will  not  be  dow  ngraded  for  that  particular  discrepancy.  An  example  of 
this  trend  was  offered  by  a  senior  Navy  acquisition  executive  w  ho  identified  two  offerors, 
both  of  whom  were  delinquent  on  their  last  contract.  The  first  offeror  maintained  the 
reason  for  delinquency  was  late  Government  Furnished  Material  (GFM).  The  second 
offeror  also  maintained  the  reason  for  delinquency  was  late  GFM,  however  the  second 
offeror  went  further  by  outlining  the  actions  taken  by  the  contractor  in  an  attempt  to  work 
around  the  problem.  The  anecdote  was  ended  with  a  question  "Based  solely  on  the 
example,  which  offeror  would  you  select?" 
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b.  Responses 


This  a  delicate  area.  In  either  case,  whenever  past  performance  is  good  or  bad,  it 
will  be  gi\’en  110  percent  of  the  evaluation  by  the  technical  e\aluators.  You  must  be 
careful  not  to  assign  an  inordinate  amount  of  weight  to  past  performance  since  it  could 
amount  to  a  responsibility  determination.  The  key  is  to  ensure  contractors  understand 
past  performance  is  important  and  tell  them  so  in  the  solicitation.  It  needs  to  be  one  of 
the  foremost  things  in  Uieir  minds  while  performing  a  contract. 

We  call  for  past  performance  data  in  the  RFP.  Contractors  submit  36-60  months  of 
prior  performance  data.  We  evaluate  this  performance  data  along  with  the  same 
information  we  get  from  the  contract  administration  office.  It  can  equate  to  a  damming 
•  (poor)  mark  if  we  find  out  poor  past  performance  by  ourselves  and  it  was  not  presented 
by  the  contractor.  A  contractor  can  fix  the  stigma  of  poor  past  performance  by 
presenting  a  management  plan  in  enough  detail  that  tells  us  how'  they  intend  to  correct 
the  deficiencies. 

Past  performance  is  a  management  factor.  Call  past  references.  If  you  only  get  one 
bad  reference  or  no  references,  then  subtract  points  for  past  performance. 

That  is  a  tough  question.  Corporate  experience  with  regards  to  products  is  much 
easier.  It  is  more  difficult  for  services  because  we  cannot  normally  say  something  like, 
"Well  85  percent  of  your  products  were  on  time  without  deficiencies"  for  service 
contracts.  How  to  evaluate  past  performance  must  be  written  into  the  SSP.  For  this 
acti^•it>’,  past  performance  is  normally  written  in  as  a  major  category. 

You  need  to  be  careful  you  do  not  cross  the  line  into  a  responsibility  issue.  For 
instance,  not  asking  for  information  on  how  the  offeror  would  complete  the  effort 
under  this  solicitation  and  concentrating  instead  on  past  performance  on  other  contracts. 

Past  performance  is  a  touchy  area  since  you  can  get  into  responsibility  issues.  It 
could  also  give  you  false  readings.  For  instance,  there  may  be  a  different  management 
team  for  this  contract  than  for  one  that  was  completed  previously. 

,  Serv  ices  are  not  yet  incorporated  in  the  contractor  performance  assessment  rating 
system.  We  use  questionnaires  to  collect  data. 

The  plan  on  how  to  evaluate  past  p)erformance  must  be  tailored  to  each  buy.  We 
look  at  how  responsive  headquarters  w'as  to  local  management  problems.  Past 
performance  also  needs  to  be  reviewed  in  light  of  what  type  of  contract  was  let  in  the 
past.  For  instance,  you  need  to  evaluate  past  performance  differently  under  a  fixed 
price  effort  compared  to  cost  reimbursement. 

6 .  Question  #6 

What  are  some  of  the  problems  or  issues  your  activity  faces  with  "best  value" 
evaluations  for  engineering  and  technical  serv'ices  contracts? 
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a.  Discussion 


The  personnel  interviewed  were  unconstrained  by  the  researcher  and 
encouraged  to  provide  as  much  information  as  they  desired.  Many  interv  iewees  provided 
three  or  four  problems,  while  others  provided  only  one.  Several  interviewees  expounded 
by  providing  a  description  of  a  problem  and  an  example  while  others  provided  only  one 
line  descriptions.  The  severity  of  the  problem  in  relation  to  the  overall  process  was  not 
assessed. 

Overall,  responses  to  this  question  were  varied  but  generally  focused  in  the 
areas  of  acquisition  planning  and  proposal  evaluation.  Although  rarely  mentioned 
specifically,  a  perv'asive  problem  perceived  by  the  researcher  was  a  lack  of  time  and/or 
resources  to  properly  plan  and  conduct  a  "best  value"  evaluation. 

The  list  of  problems  is  segregated  into  three  categories;  1)  Acquisition 
Planning,  2)  Proposal  Evaluation,  and  3)  Other.  A  brief  description  of  the  problem  and  the 
number  of  times  it  was  mentioned  is  listed  below  each  category.  Problems  mentioned  most 
often  are  further  defined  and  anal>’zed  in  Chapter  IV. 

CATEGORY 

Acquisition  Planning  Times  Noted 

1 .  Requiring  acti\'ity  feels  constrained  by  a  5 

technical/cost  tradeoff  during  "best  value"  evaluations. 

Instead,  the  evaluation  should  focus  entirely  on  the 
technical  proposal. 


2.  Improper  planning  of  evaluation  criteria  used  in  10 

proposal  evaluation  to  differentiate  between  offerors. 

3 .  PWSs  written  restrictively  allowing  unfair  advantage  4 

to  incumbent. 

4.  "Best  value"  method  employed  when  low  price/technically  5 

acceptable  offeror  strategy  would  yield  the  same  result. 

5.  Too  many  subfactors  add  complexity  to  process  and  3 

dilute  results. 
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6.  Poor  communications  between  procuring  and  requiring  6 

activities  in  acquisition  planning  phase. 

7 .  Ex’aluation  criteria  established  which  do  not  deal  with 

true  quality  considerations  between  offerors.  2 

Proposal  E\'aluation 

1.  Contractor  gaming  on  cost  and  technical  proposals.  19 

2.  The  technical  evaluation  committee  is  not  trained  nor  19 

is  it  sensitive  to  the  proposal  evaluation  process. 

3 .  SSP  formulated  by  different  group  other  than  personnel  2 

who  actually  conduct  the  technical  evaluation. 

4.  Problems  with  past  performance  data  use  in  evaluations;  6 

1)  using  past  performance  in  evaluations  as  a  guise  for 


responsibility'  determinations,  2)  fairness  issues  associated 
with  using  past  performance  to  downgrade  a  technically 
acceptable  offeror  with  no  experience,  and  3)  evaluating  past 
performance  on  different  management  teams  within 
the  same  corporation. 

5.  Contractor  bait  and  switch  tactics  with  key  personnel.  4 

6.  Resume  requirements  v^'ritten  and  evaluated  for  specified  4 

personnel,  i.e.,  personal  sen  ices. 

7.  Cost  realism;  only  a  paucity  of  guidance  available  on  4 

how  to  complete. 

Other  Issues 

1 .  Contractors  distrust  the  "best  \  alue"  source  selection  1 1 

method.  "Best  value"  represents  a  method  the 

Government  uses  to  select  the  contractor 
they  wanted  in  the  first  place. 

2 .  Government  sending  the  incorrect  signals  to  contractors  3 

by  cutting  budgets,  yet  continuing  to  offer  premiums. 

3.  Legal  considerations.  "Best  value"  evaluations  offer  too  2 

many  areas  to  protest. 

4.  Centralization  of  contracting  activities,  NRCCs,  is  not  3 

an  organization  conducive  to  obtaining  the  "best  value". 

5.  Contract  administration:  The  Go>'emment  is  not  getting  5 

what  it  is  paying  for  in  higher  rated  technical  proposals. 


41 


5 


6.  Too  many  changes  to  the  procurement  system.  Policymakers 
attempting  to  transform  an  inherently  subjecli^■c 
process  into  an  objectix  e  one. 

7.  Funding:  Requiring  activities  limit  requirements  based  on  2 

funding  constraints.  When  more  funds  become  a\'ailablc 

requiring  acti\  ities  desire  to  increase  scope  of  contract. 

7 .  Question  #7 

How  does  your  activity  determine  the  need  to  pay  a  premium  for  a  technically 
superior  proposal  over  a  proposal  which  is  technically  acceptable,  yet  lower  priced? 
a.  Discussion 

The  essence  of  the  "best  value"  concept  lies  in  the  reasoned  judgment  of  the 
SSA  in  assessing  relative  strengths  and  weaknesses  of  competing  proposals.  In 
determining  whether  to  pay  a  premium  for  a  higher  rated  technical  proposal,  the 
interviewees  resoundingly  emphasized  the  use  of  this  judgment.  Of  course,  in  order  to 
assist  in  making  the  judgment  more  educated,  many  interviewees  stressed  the  importance  of 
a  cost  benefit  tradeoff  analysis  which  assesses  the  impact  to  the  Government  resulting  from 
a  technically  superior  proposal  with  the  highest  score  or  rating,  in  comparison  to  the 
proposal  which  is  technically  inferior  but  provides  the  low  er  evaluated  cost.  This  tradeoff 
analysis  is  performed  by  the  technical  evaluation  committee  w  ho  assesses  and  documents 
the  technical  differences  in  a  quantitative  format.  Essentially,  they  determine  how  much 
more  the  technical  superiority  is  worth  in  dollars? 

A  more  objective  measure  of  quantifying  the  cost/technical  tradeoff 
eiKXHintered  during  this  study,  was  in  the  form  of  a  linear  relationship  w  here  the  acquisition 
planning  process  established  a  minimum  acceptable  technical  score.  Additional  technical 
points  over  the  minimum  acceptable  score  are  equated  with  a  percentage  premium  the 
activity  would  be  willing  to  pay.  For  instance,  if  the  minimum  acceptable  technical  score 
was  80  on  a  100  point  scale  and  technical  was  tw'o  times  more  important  than  cost,  the 
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actiN  it}’  would  be  w  illing  to  pa}  up  to  a  40  percent  premium  o\  er  the  lowest  acceptable 
technical  proposal,  or  cost  for  each  point  in  technical  above  the  low  acceptable  offeror. 
This  premium  is  advertised  in  the  solicitation.  It  should  be  emphasized  that  even  with  this 
objecti\’e  measure,  personnel  at  the  activity  stressed  that  the  use  of  judgment  continues  to 
be  the  ke}  to  the  tradeoff  process. 

One  final  area  w  here  reasoned  judgment  plays  a  key  role  is  when  technical 
scores  are  ver}’  close.  If  this  occurs,  a  determination  may  be  made  to  call  the  technical 
proposals  equal  and  the  contract  award  is  made  on  the  basis  of  lowest  e\  aluated  cost. 
h.  Responses 

This  is  not  really  a  big  problem.  We  use  a  premium  method  which  represents  a 
linear  relationship  between  cost  and  technical.  For  instance,  one  point  of  technical  is 
worth  two  percent  of  a  cost  premium.  If  score  is  within  one  or  two  points,  we'll 
usually  call  the  proposal  equal  on  technical  merit  and  award  to  the  low  offeror. 

We  try  to  use  the  best  engineering  and  technical  judgment.  As  an  SSA,  I  require 
the  chairman  of  the  SSEB  to  explain  differences  to  me  in  terms  I  can  understand.  If  I 
can  understand  the  differences,  then  other  people  can  too.  I  call  it  the  reasonableness 
test. 

You  need  to  look  at  the  discriminators  which  were  set  up  in  the  SSP  and 
communicated  to  offerors  via  the  solicitation.  What  are  the  differences  in?  There  is  no 
formula  for  it  It  all  depends  on  deliberative  and  reasoned  judgment. 

Mathematical  formulations  to  arrive  to  this  point  are  only  tools.  We  are  not 
compelled  to  award  to  an  offeror  based  on  high  score  alone.  The  contracting  officer 
and  chairman  of  the  SSEB  present  all  the  facts  to  the  SSA  who  utilizes  educated 
judgment. 

This  happens  frequently.  You  need  to  go  back  to  the  SSEB  and  have  them  tell  you 
how  much  the  extra  techni<^  is  worth  in  dollars.  They  (technical  evaluators)  do  not  get 
the  difference  in  dollars  up  front.  They  could  possibly  manipulate  the  system  if  they 
had  the  costs. 

We  require  a  cost  impact  statement  from  technical  which  basically  equates  to  a 
cost/price  tradeoff  analysis.  You  can  give  them  costs  before  hand  or  ask  them  to  justify 
an  amount  the  extra  technical  is  worth.  If  technical  is  very  close,  based  on  narrative, 
we'll  go  with  the  low  cost  offeror. 
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There  is  no  specific  guidance  on  the  mechanics  of  the  tradeoff  analysis  since  we  do 
not  want  to  tie  the  hands  of  the  SSA.  I  have  seen  it  done  both  ways.  Supplying 
technical  w  ith  the  costs  at  this  point  and  not  supplying  them  w  ith  costs.'  Whate\  er  the 
method,  the  narrati\  e  is  critical. 

The  chairman  of  the  SSEB  has  to  be  able  to  communicate  that  for  this  much  more  in 
price  these  are  the  advantages  we  will  obtain  so  that  a  Colonel  or  General  (SSA)  will 
understand. 

It  all  boils  down  to  a  judgment  call  by  the  SSA  w  hich  is  not  as  objective  as  some 
might  be  led  to  believe. 

8.  Question  #8 

How  does  your  acti\ity  conduct  the  cost  realism  analysis  for  engineering  and 
technical  ser\  ices  contracts? 
a.  Discussion 

Responses  to  this  question  emphasized  the  use  of  independent  Government 
cost  estimates  based  on  historic  and  similar  efforts.  Should  cost  and  most  probable  cost 
estimates  were  also  mentioned  utilizing  DCAA  provided  labor,  overhead  and  general  and 
administrative  rates,  the  Department  of  Labor's  wage  determinations  and  Goxemment  civil 
service  wage  scales  as  standards.  Overall,  responses  typically  described  the  cost  realism 
analysis  as  a  time  consuming  process  many  times  exacerbated  by  company  ventures 
requiring  the  use  of  subcontractors  whose  costs  are  also  subject  to  analysis. 

Examples  of  cost  realism  analyses  reviewed  at  several  activities  revealed  that 
they  concentrated  on  the  major  cost  elements;  direct  labor,  overhead,  and  general  and 
administrative  expenses.  Material,  other  direct  charges  and  escalation  rates  are  usually 
"normalized"  for  engineering  and  technical  sen  ices  contracts  meaning  that  there  should  be 
no  logical  differences  between  contractors  on  these  costs.  As  a  result,  all  offerors  are 
evaluated  on  the  same  normalized  cost  figures. 

Direct  labor  rates  were  usually  compared  to  DCAA  audited  labor  rates  and  rates 
from  previous  or  similar  contracts.  For  the  Service  Contract  Act  covered  labor  categories, 
there  is  normally  a  comparison  of  proposal  rates  with  those  published  by  the  Department  of 
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Labor.  Of  course,  comparisons  of  rates  are  not  always  straight  fonvard.  At  times,  labor 
categories  pror  ided  b\’  the  Go^•emment  do  not  match  the  nomenclature  of  categories 
provided  by  the  contractor.  Section  L  of  the  RFP  normally  requires  a  cross  reference  to 
match  labor  categories  represented  in  the  RFP  to  those  proposed. 

Overhead  rates  were  compared  to  DCAA  audited  overhead  rates  and  overhead 
rates  on  previous  contracts.  Several  interviewees  noted  that  there  was  a  "twist"  to 
comparing  overhead  rates  in  this  type  of  analysis  where  the  objective  is  to  ensure  that  the 
proposal  is  high  enough  to  cover  the  cost  of  the  contract  performance.  This  is  in  contrast  to 
DCAA's  normal  concern  which  is  to  ensure  that  the  Government  is  not  being  overcharged. 
Fringe  benefits  is  another  key  concern  when  evaluating  overhead  rates.  The  concern  is 
with  the  ability  of  a  contractor  to  retain  quality  employees  w'ithout  providing  adequate 
fringe  benefits  such  as  vacations  and  medical  plans.  DCAA  audits  and  historical  costs  are 
also  consulted  with  regards  to  uncomjjensated  overtime,  which  can  be  used  as  an  indicator 
of  the  expected  quality  of  w  ork  and  retainability  of  employees. 

Opinions  from  interv  iewees  and  a  rev  iew  of  a  small  sample  of  actual  cost 
realism  analyses  indicated  that  proposal  costs  are  adjusted  upward  on  a  large  percentage  of 
proposals.  In  fact,  one  interviewee  estimated  that  costs  were  adjusted  upw'ard  at  least  75 
percent  of  the  time  based  on  a  cost  realism  analysis.  This  estimate  may  be  on  the  upper  end 
of  the  spectrum,  but  as  evidenced  from  the  number  of  respondents  noting  contractor 
gaming  on  costs  as  a  problem,  the  percentage  of  upward  adjustments  of  contractor  costs  is 
relatively  large. 

b.  Responses 

Labor  rates  are  the  important  things  in  this  type  of  contracting.  You  really  need  to 
look  at  and  compare  expertise  and  quality  with  the  rates  proposed. 
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Contractor  gaming  makes  cost  realism  difficult.  Companies  form  joint  ventures 
with  others  where  nobody  with  resumes  proxided  is  assigned  to  the  company  being 
evaluated.  Companies  set  up  new  cost  centers.  There  is  reall\  no  one  way  to  do  cost 
realism  and  answ  er  all  the  questions. 

This  activity  uses  SCA  dollar  wage  advisories  and  historic  rates.  Basically  a 
thorough  should  cost  analysis. 

It  is  difficult  to  find  material  on  cost  realism.  Each  person  does  it  different.  For 
this  contract,  I  developed  my  own  approach  establishing  a  matrix  including  past 
contract  information  and  rates  from  a  similar  contractor  in  the  area. 

Cost  realism  requires  balancing  the  cost  proposal  and  the  technical  proposal  to 
ensure  things  match  up.  Normally  it  does  not  require  the  assistance  of  the  technical 
people  to  do  this. 

We  get  very  in\  ol\  ed  in  the  personnel  costs  proposed  to  ensure  these  costs  match 
the  technical  proposal.  There  have  been  instances  where  the  entire  proposal  has  been 
refuted  on  a  mismatch  benveen  the  two. 

Many  contractors  are  reliable,  others  are  not.  Lots  of  extra  work,  time,  and  effort 
are  expended  on  cost  realism  because  you  cannot  rely  on  their  (contractors')  data. 

Since  we  do  not  always  ha\'e  an  independent  Government  cost  estimate,  we  rely  on 
a  most  probable  cost  scenario  where  the  customer  provides  an  estimated  labor  rate, 
labor  estimate,  and  materials. 

Our  job  is  to  figure  out  if  costs  proposed  are  realistic,  fair,  and  reasonable,  and 
there  are  too  many  intricacies  to  making  up  specific  guidelines  on  the  subject. 

Cost  realism  analysis  require  saavy  and  experience. 

Cost  realism  requires  ethical  behavior  from  both  the  contractor  and  the 
Government. 


C.  CONTRACTOR  QUESTIONNAIRE 
1 .  Question  #1 

How'  would  you  define  "best  (greatest)  value"? 
a.  Discussion 

Definitions  provided  by  contractors  were  completely  consistent  with  definitions 
provided  by  Government  personnel.  A  total  of  seven  contractors  were  interviewed  for  this 
study,  and  all  were  familiar  with  the  term  and  the  concept.  All  provided  serv  ices  to  the 
Navy. 
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2 .  Question  #2 

How  do  you  feci  about  the  "best  value"  concept  utilized  by  the  Government  in 
awarding  contracts? 

a.  Discussion 

Consensus  among  the  sample  of  contractors  contacted  for  this  study  believed 
that  the  "best  value"  concept  is  a  good  one  for  two  reasons.  First,  its  premise  is  based  on 
high  quality  over  low  price,  and  second,  because  it  is  a  method  of  protecting  responsible 
offerors  from  offerors  w  ho  bid  unrealistically  low  rates  w'hen  competition  is  based  on  price 
alone,  i.e.,  a  $7  per  hour  engineer.  A  program  manager  for  a  major  support  contractor  that 
was  recently  awarded  a  "best  value"  contract,  admitted  that  he  knew  his  firm  was  not  the 
low  bidder  after  the  award.  He  also  admitted  that  due  to  the  "best  value"  aw'ard,  he  feels 
the  firm  is  more  motivated  to  perform  quality  w'ork  rather  than  focusing  on  lowering  costs. 

Unfortunately,  contractors  do  no  always  beliexe  that  the  Government  is 
focusing  on  quality  in  lieu  of  costs.  This  was  mentioned  by  both  Government  and 
contractor  personnel.  Instead,  contractors  accuse  Government  procurement  officials  of 
being  overly  concerned  with  the  low  bid.  Some  of  this  perception  is  true.  One 
Government  interviewee  admitted  how  he  was  unable  to  eliminate  a  technically 
unacceptable  offeror  from  the  competitive  range  because  "the  price  was  too  good." 

The  perception  held  by  contractors  of  the  Government  being  concerned  solely 
with  price,  leads  to  cost  and  technical  gaming  sometimes  attributed  to  "remaining 
competitive"  and  "fighting  fire  with  fire."  As  one  contractor  stated,  "Gaming  is  a  result  of 
the  rules  of  the  road  and  contractor's  mistrust  of  the  process.  I  feel  Government  should 
award  on  the  basis  of  technical  superiority  only!" 
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b.  Responses 


I  do  not  beUe\'e  Go\  emmenl  is  always  buying  the  "best  value".  Everyone  needs  to 
change  their  mindset  before  "best  value"  will  happen.  The  current  mindset  is  on  low 
cost. 

I  think  it  is  great!  There  have  been  a  great  proliferation  of  companies  bidding  jobs 
they  cannot  do.  "Best  \  alue"  is  a  wa)'  to  correct  it. 

"Best  value"  is  easier  to  do  with  something  tangible  like  widgets.  It  is  more 
difficult  in  a  softer  area  like  sen  ices.  With  widgets  you  can  look  at  NTTTF,  warranty 
and  other  things. 

"Best  value"  is  the  only  way  to  purchase  something  that  is  not  under  detailed 
specifications.  Unfortunately,  pressure  is  on  the  Government  contracting  officer  to 
award  to  the  low  offeror.  The  Government  can  do  a  better  job  in  this  area.  Too  many 
contracts  are  awarded  to  the  low  bidder. 

The  concept  is  right  on  because  picking  contractors  based  on  the  low  bid  is  no 
good.  Under  low  bid  you  get  the  $7.00  per  hour  engineer  and  firms  have  to  fight  fire 
with  fire.  I  sometimes  think  "best  value"  is  only  lip  ser\'ice  though. 

"Best  value"  is  terrific  if  the  Government  can  stick  to  it. 

3 .  Question  #3 

Does  your  firm  have  any  problems  with  "best  value"  solicitations  let  by  DoD  for 
engineering  and  technical  sen  ices?  If  so,  what  are  they? 

a.  Discussion 

The  responses  to  this  question  presented  an  almost  even  split  of  contractors 
who  believe  "best  value"  solicitations  are  clear  and  those  who  do  not.  Half  of  the 
respondents  believed  that  solicitations  offered  ample  information  regarding  the  evaluation 
criteria  and  what  w  as  important  to  the  Government.  The  other  half  do  not  believe  that  the 
evaluation  criteria  are  clear  partly  due  to,  as  one  contractor  stated,  "consistently  changing 
language  in  Sections  Land  M  of  the  RFP."  Although  procuring  activities  have  attempted  to 
standardize  evaluation  factors  and  "best  value"  wording  in  RFPs,  there  is  little 
standardization  betw'een  different  activities.  Compounding  the  confusion  are  the  different 
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rating  systems  used  at  different  acti\  itics.  Contractors,  constrained  by  resources  and  faced 
with  what  some  term  "cut  throat"  competition,  need  to  determine  where  to  place  their 
emphasis  when  constructing  a  proposal. 
b.  Responses 

Go^■emment  tries  to  define  pre-established  rates  and  levels  of  qualifications.  Man\' 
times  they  (Government)  fail  on  both  counts. 

Past  performance  if  e^•aluated  is  difficult  for  the  new  guy  on  the  block.  You  may 
have  a  new  guy  who  is  great .  You  penalize  him  because  of  little  past  performance  data. 

I  am  never  really  clear  what  the  evaluation  criteria  are.  NASA  does  a  good  job  in 
this  area,  however,  DoD  is  somewhat  haphazard.  Good  criteria  means  it  is  easier  to 
understand  w'hat  the  Government  will  evaluate. 

I  have  no  problems  with  the  solicitations.  What  I  do  have  problems  with  is  that  I 
lack  confidence  that  the  Government  is  actually  going  to  award  "best  value".  There  are 
problems  in  the  environment.  Rules  continue  to  change.  It  used  to  be  low'  offeror  with 
discussions,  now  it  is  "best  value"  w'ithout  discussions.  Sometimes  it  is  hard  to  keep 
up. 


4.  Question  #4 

How  do  you  feel  about  awarding  contracts  on  a  "best  value"  basis  w'ithout 
discussions? 

a.  Discussion 

Contractors  and  Government  personnel  favor  awarding  contracts  without 
discussions  because  it  can  save  resources  by  streamlining  the  process.  However,  in 
responding  to  this  question,  all  contractors  cautioned  that  awarding  without  discussions  is 
good  policy  only  when  PWSs,  instructions  to  the  offeror,  and  evaluation  criteria  are  well 
defined.  These  criteria  are  typically  not  present  for  engineering  and  technical  services 
unless  the  effort  is  mature  and  well  established.  This  researcher  encountered  only  one 
engineering  and  technical  service  contract  awarded  without  discussions.  It  was  awarded  to 
an  incumbent  who  was  evaluated  w  ith  the  high  technical  score  and  low  cost.  Furthermore, 
in  this  situation,  it  w'as  believed  that  no  other  competitor  w'as  susceptible  to  award  of  the 
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contract.  However,  a  super\  isor%’  contract  specialist  at  one  acti\  ity  estimated  that  up  to 
25%  of  the  contracts  awarded  at  his  activity  are  awarded  without  discussion  to  other  than 
the  low  priced  offeror. 

5 .  Question  #5 

In  your  view ,  does  the  Go\  emment  do  an  adequate  job  of  cost  realism  analysis? 
a.  Discussion 

Two  contractors  cited  the  Government's  poor  histoiy-  in  performing  a  cost 
realism  analysis  through  the  example  of  the  $7  per  hour  engineer.  However,  at  present,  the 
overwhelming  majority  believe  that  the  Government  is  performing  more  accurate 
assessments  of  contractors'  actual  costs.  Moreover,  contractors  now  believe  that  the 
Government  is  comparing  cost  and  technical  proposals  for  cost  realism  purposes. 

Of  course  there  are  still  a  number  of  contractors  w'ho  doubt  the  effectiveness  of 
a  cost  realism  analysis.  One  contractor  responded  that  he  sees  "a  lot  of  high  sounding 
ivords  about  cost  realism  in  RFPs,  but  I  sometimes  believe  it  is  only  lip  serv  ice,"  citing  an 
example  where  his  firm  was  the  low  price  offeror  prior  to  the  Government  adjusting  their 
proposal  price  upw  ard.  He  felt  that  his  firm  lost  the  contract  largely  because  of  this  price 
adjustment  and  stated  that  after  such  an  occurrence  "it  always  makes  you  wonder  if  the 
Government  is  only  justifying  a  w-ay  to  pick  the  contractor  it  w'anted  in  the  first  place." 
Another  contractor  responded  that  "industry  knows  more  about  its  costs  than  does  the 
Government,"  raising  the  issue  of  industry's  opportunity  to  justify  its  cost  prior  to  award 
or  request  for  a  BAFO. 

Finally,  every  contractor  interviewed  brought  up  the  topic  of  uncompensated 
overtime  (UCOT)  while  discussing  the  cost  realism  analysis.  All  contractors  in  this  small 
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sample  bclie\  e  that  UCOT  usage  in  proposals  should  be  prohibited  by  regulation  since  it 
allegedh'  "destabilizes  the  playing  field".  The  UCOT  issue  is  further  defined  and  addressed 
in  Chapter  IV. 

b.  Responses 

(Cost  realism)  Used  to  be  horrendous.  Resumes  of  the  best  people  were  in  the 
technical  proposal  but  not  in  the  cost  proposal.  UCOT  problems-  how  much  UCOT  is 
actually  being  applied  to  the  contract  in  question?  Before  the  technical  and  cost  people 
never  got  together.  Today,  I  think  that  is  different.  People  are  starting  to  talk  to  one 
another  in  my  estimation. 

UCOT  is  still  a  problem  judging  by  input  from  the  field,  but  it  is  now  popping  up  in 
different  places  like  Hunts\’ille,  Alabama. 

Sometimes  the  contractor  has  a  better  understanding  of  what  the  effort  will  cost  than 
does  the  Government.  We  have  more  years  of  experience. 

On  our  current  contract  yes.  The  Government  should  watch  UCOT  though  because 
in  my  estimation  it  leads  to  a  lower  quality  of  work. 

Government  does  a  good  job  if  they  want  to.  Problems  in  evaluating  cost  realism 
are  in  the  area  of  UCOT  which  is  voodoo.  Even  those  who  bid  it  do  not  understand  it. 
UCOT  is  screwing  over  the  Government  and  the  contractors.  It  is  hard  to  justify  and 
cost.  It  forces  Government  to  look  at  the  bottom  line. 


D.  SUMMARY 

Chapter  III  discussed  and  presented  responses  to  a  survey  questionnaire  used  to  solicit 
responses  from  Government  and  industry  personnel.  Data  collected  from  a  sample  of 
SSPs,  RFPs  and  SSDMs  were  also  utilized  to  respond  to  several  questions  in  Chapter  III. 
Major  problems  and  issues  discovered  as  a  result  of  the  survey  questionnaire  are  the  subject 
of  Chapter  IV. 
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IV.  PROBLEM  DEFINITION  AND  ANALYSIS 


A.  INTRODUCTION 

This  chapter  will  focus  on  the  definition  and  analysis  of  the  most  frequently  mentioned 
problems  associated  with  "best  value"  evaluations  identified  in  Chapter  III.  The  most 
frequently  mentioned  problems  discussed  in  this  chapter  are:  1)  the  selection  and  weighting 
of  eN’aluation  criteria,  2)  the  technical  evaluation  process,  3)  contractor  gaming  in  cost  and 
technical  proposals  and  4)  contractor  distrust  of  the  "best  value"  source  selection  process. 

B.  'PROBLEM  #1:  EVALUATION  CRITERIA 

The  determination  and  weighting  of  appropriate  evaluation  criteria  which  differentiate 
between  offerors  for  use  in  "best  value"  evaluations. 

1 .  Problem  #1  Defined 

The  determination  of  the  proper  evaluation  criteria  for  use  in  source  selection  is  a 
critical  aspect  of  the  acquisition  planning  process.  It  was  mentioned  as  a  problem  by 
several  respondents  to  the  research  questionnaire  as  well  as  by  NAVSUP,  which  noted 
Problem  #1  as  an  area  of  concern  from  both  the  technical  and  procurement  personnel.  This 
was  documented  in  a  report  by  the  NaN'al  Supply  Systems  Command  Greatest  (Best)  Value 
Source  Selection  Process  Action  Team  which  stated  a  "Need  for  the  joint  development  of 
source  selection  plans  which  lead  to  differentiation  between  offerors  and  emphasis  on  past 
performance  and  quality"  [Ref.  15:p.  4).  Problems  in  this  area  were  also  observed  during 
visits  to  contracting  activities. 


52 


2 .  The  Importance  of  Evaluation  Criteria 

E\  aluaUon  criteria  are  essential  in  pro\  iding  a  framew  ork  necessar)’  to  ensure  that  a 
quality  service  is  obtained.  The  SSP  is  used  to  communicate  these  criteria  plainly  to  source 
selection  officials  and  is  the  crucial  document  in  preparing  the  RPT  which  communicates 
these  criteria  to  potential  offerors. 

It  is  important  that  the  evaluation  criteria  be  individually  tailored  to  each 
procurement  in  accordance  with  FAR  15.605.  The  evaluation  criteria  should  also  be 
limited  so  as  to  select  criteria  which  will  focus  on  the  important  issues  for  the  successful 
accomplishment  of  the  PWS.  Proper  selection  of  evaluation  criteria  is  necessary  to 
effectively  discriminate  among  the  proposals  and  perniit  the  correct  ranking  of  the 
proposals  in  the  order  of  their  value  to  the  Government. 

Properly  identifying  the  evaluation  criteria  increases  the  probability  of  evaluating 
and  selecting  the  most  advantageous  contractor.  If  the  requiring  activity  knows  what 
aspects  an  offeror's  proposal  should  address  and  how  these  aspects  will  be  evaluated  prior 
to  the  technical  evaluation,  then  chances  are  much  greater  of  conducting  a  disciplined 
source  selection  and  making  the  most  advantageous  procurement  decision.  If  the 
evaluation  criteria  are  poorly  identified,  there  may  exist  inconsistencies  in  the  technical 
evaluation  process  which  increases  the  time  required  to  perform  the  evaluation  and  the 
possibility  of  protest  by  unsuccessful  offerors. 

3 .  Deficiencies  Mentioned  by  Contracting  Personnel  Regarding  the 

Evaiuation  Criteria  in  "Best  Value"  Procurements 

The  following  comments  collected  during  this  research  reflect  the  problems  with 
evaluation  criteria  for  use  in  "best  value"  procurements: 

We  utilize  14  standard  factors  and  subfactors  to  evaluate  offerors.  Unfortunately, 
this  sometimes  has  an  averaging  effect. 
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It  is  sometimes  hard  to  discriminate  between  offerors  of  engineering  and  technical 
serv  ices.  Sample  tasks  are  a  good  way  to  do  so  if  the\’  are  reasonable  and  e\aluation  of 
them  is  critical. 

E\’aluation  factors  (identified  by  the  requiring  activity)  are  not  always  intrinsic  to  the 
job.  For  example,  does  it  really  matter  if  the  contractor's  facility  is  close  by  or  far 
away?  Sometimes  it  makes  no  difference. 

Using  the  right  evaluation  criteria  is  "motherhood"  for  all  procurements.  They 
(evaluation  criteria)  are  usually  harder  to  write  for  "best  value"  procurements  though. 

One  of  our  biggest  concerns  is  to  ensure  we  have  finite  discriminators  and  actual 
tasks  which  are  needed  to  be  performed.  For  instance,  in  the  area  of  resumes  for  key 
personnel,  we  ha\e  to  watch  out  since  we  are  buying  a  level  of  expertise  and  not 
personal  ser\  ices. 

The  hard  part  about  "best  value"  is  setting  up  evaluation  criteria  which  will  get  you 
there. 

Several  times,  (evaluation)  criteria  made  up  by  the  requiring  activity  do  not  make 
common  sense.  Why  do  you  need  someone  with  that  much  experience? 

One  has  to  be  careful  not  to  lose  the  forest  through  the  trees  when  identifying 
evaluation  criteria.  It  is  really  important  for  technical  to  understand  their  requirement 
and  what  they  consider  important  to  accomplish  it.  Many  times  technical  chooses  too 
many  criteria. 

4.  Examples  of  Problem  Situations  with  Evaluation  Criteria 
Encountered  During  this  Research 

Examples  of  problems  with  "best  value"  evaluation  criteria  encountered  during 
research  are  presented  below .  Each  example  demonstrates  how  the  selection  of  evaluation 
criteria  can  affect  the  procurement  of  the  most  advantageous  procurement  decision.  The 
first  example  presents  the  standard  criteria  and  subcriteria  used  in  evaluations  at  one  activity' 
which  at  times  has  the  effect  of  diluting  evaluation  results.  Diluting  results  means  it  is  more 
difficult  to  discriminate  between  offerors.  The  second  example  demonstrates  how  too 
many  evaluation  criteria,  poorly  presented  can  complicate  the  technical  evaluation  process 
and  make  it  more  difficult  to  determine  the  most  advantageous  procurement  decision.  The 
final  example,  presents  technical  criteria  discussed  at  a  meeting  attended  by  the  researcher 
between  a  requiring  activity  and  contracting  personnel  regarding  evaluation  criteria  for  a 
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"best  value"  prcx:uremeni.  As  a  result  of  the  meeting,  the  acquisition  strategy  changed  from 
"best  value"  to  a  low  cost  technicalh  acceptable. 
a.  Example  #7 

The  following  are  technical  evaluation  criteria  and  subcriteria,  quoted  from  a 
standard  source  selection  plan  used  at  one  activity.  These  criteria  sometimes  have  the 
potential  effect  of  diluting  the  results  of  the  technical  evaluation.  The  sample  plan  states 
that  the  factors  used  in  the  technical  evaluation  "shall  include  all  the  evaluation  factors 
described  below  ;" 

1 .  CRITERIA 

a.  Technical/  Management 

( 1 )  Technical  Approach  (Criteria) 

Subcriteria: 

(a)  Understanding  and  Approach 

(b)  Specific  Techmeal  Capability 

(c)  Problem  Areas 

(d)  Unique  Ideas  and  Capabilities 

(e)  Workforce  Loading  Plan 

‘  (0  Start-Up  Plan 

(g)  Sample  Tasks 

(2)  Experience  (Criteria) 

Subcriteria; 

(a)  Personnel  Qualifications 

(b)  Corporate  Experience 

(3)  Management  Approach  (Criteria) 

Subcriteria; 

(a)  Contract  Management 
(include  subcontractors,  if  any) 

(b)  Interface 

(c)  Organization 

(include  subcontractors,  if  any) 

(d)  Cost  Management 

(4)  Facilities  and  Resources  (Criteria) 

No  subcriteria 
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The  problem  with  employing  the  abo\e  number  of  criteria  and  subcriteria  to 

evaluate  proposals  is  the  potential  averaging  effect.  In  total,  there  are  four  criteria  and  13 

subcriteria  for  utilization  during  a  technical  evaluation.  Attaching  weights  to  each  of  the 

above  criteria  and  subcriteria  dilutes  the  effect  of  what  may  really  be  important  to  the 

requiring  activit}'  and  therefore  makes  it  more  difficult  to  discriminate  between  offerors.  In 

this  situation,  requiring  activities  should  be  afforded  the  opportunity  to  choose  only  the 

most  important  criteria  for  the  evaluation.  All  criteria  and  subcriteria  may  not  require 

review  in  each  evaluation. 
b.  Example  U2 

This  example  demonstrates  how  too  many  criteria,  poorly  presented,  can 
eomplicate  the  technical  evaluation  and  increase  the  time  required  for  the  source  selection 
process.  The  result,  limits  the  effectiveness  of  a  "best  value"  evaluation  since  technical 
evaluators  may  not  be  certain  as  to  which  considerations  are  the  most  important  to  the 
requiring  actiN  ity.  In  this  example,  there  are  far  too  many  criteria  or  "considerations"  to 
review  in  completing  the  technical  evaluation.  In  addition,  there  are  no  weights, 
quantitative  or  qualitative  attached  to  any  of  the  considerations.  As  a  result,  it  is  not 
possible  to  determine  which  of  the  considerations  are  important  and  which  ones  are  not. 
One  evaluator  may  grade  the  factor  very  high  based  on  one  or  se\’eral  criteria,  w  hile  another 
evaluator  may  grade  the  proposal  low  based  upon  other  criteria.  The  follow  ing  is  quoted 
directly  from  the  technical  evaluation  plan. 
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PLAN  TO  ACCOMPLISH  WORK 

Maximum  Score  Available-  10.0  points 

Each  evaluator  shall  re\  iew  and  score  the  proposal  of  the  offeror  based  on  the 
following  subfactors. 

Subfactor  1:  Organization 

Maximum  Score  Available-  5.0  points 

Structure  of  the  organization,  or  proposed  organization,  and  relationships  to  the 
task  of  the  parent  companj’,  and  subcontractors  (if  any)  and  interface. 

Considerations: 

Does  the  proposal  detail  and  specify  clearly  that  the  offeror  has  a  thorough 
understanding  of  the  requirement?  Are  the  technical  problems  understood?  Are  valid  and 
practical  solutions  set  forth?  Is  there  an  organization^  structure  chart  in  the  proposal?  Do 
the  lines  of  communication  appear  clear?  Are  relationships  to  tasks  of  the  parent  company 
fully  explained?  Does  the  organizational  structure  contain  any  ambiguities  when  compared 
to  the  other  sections?  Are  lines  of  communication  clear  between  key  personnel  and 
management?  How  will  the  contractor  communicate  with  the  requiring  acti\  it\  ?  Are  the 
lines  of  communication  clear?  How  will  the  contractor  communicate  with  any  proposed 
subcontractors?  When  compared  with  resumes,  are  key  personnel  authorized  appropriate 
responsibility?  Does  the  proposal  indicate  that  the  appropriate  number  of  personnel  are 
employed  on  a  permanent  basis  and  available  during  key  workload  periods?  Is  deliveiy’ 
order  processing  discussed?  How  will  the  order  be  processed  by  the  offeror  and 
disseminated  to  key  personnel?  Are  any  contractual  arrangements  other  than  cost 
reimbursement  contemplated  for  this  proposal?  Are  there  any  firm  fixed  price  or  time  and 
materials  type  subcontracts  proposed?  If  proposed,  is  there  a  rationale  to  document  any 
proposed  technical  advantage?  Does  the  proposal  indicate  a  start  up  plan  to  fully  perform 
within  sixty  days  after  award  of  the  contract?  Is  the  plan  realistic  and  detailed?  Does  the 
proposal  explain  the  proper  methodology  and  procedures  to  responding  to  delivery  orders? 
Do  these  plans  show  management  responsibility?  Are  uncompensated  overtime  hours 
proposed?  Is  the  appropriate  information  provided  as  indicated  in  section  L?  Are  the 
uncompensated  hours  proposed  a  substantial  benefit  to  the  Government?  How  do 
subcontractors  interact  w  ith  the  prime?  Is  this  documented  sufficiently?  Attachment  (2) 
%ts  forth  a  checklist  to  assist  the  evaluator  in  determining  the  evaluated  score. 


Evaluator's  Score _ 

Evaluator's  narrati\e: 

The  above  example  includes  27  considerations  for  each  proposal.  Some  of  the 
considerations  are  subjective,  i.e..  Do  the  lines  of  communication  appear  clear?  Other 
considerations  are  more  objective,  i.e..  Is  an  organizational  chart  presented  in  the  proposal? 
Based  on  the  number  of  considerations  and  the  mix  of  subjective  and  objective  measures,  it 
may  be  extremely  difficult  if  not  impossible  to  differentiate  between  offerors.  Which 
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offeror  presents  the  most  advantageous  procurement  decision  depends  upon  which 
considerations  the  technical  evaluator  determines  to  be  the  most  important. 

I 

In  addition  to  subfactor  #1  presented  above,  "Organization",  the  total  technical 
evaluation  consisted  of  nine  subfactors  with  approximately  20  considerations  each. 
Besides  complicating  the  evaluation  process,  the  amount  of  time  and  resources  required  to 
assess  and  document  each  proposal  evaluation  may  be  excessiv  e. 

A  less  complicated  format  for  presenting  and  rating  evaluation  criteria  for  a 
technical  evaluation  was  extracted  from  another  sample  evaluation  plan  and  is  presented 
below.  This  example  is  for  the  subfactor  "technical  approach-  sample  tasks". 

SUBFACTOR:  TECHNICAL  APPROACH 

OFFEROR _ 

EVALUATOR _ 

PART  2-  SAMPLE  TASKS 
SAMPLE  TASK# 

To  what  extent  did  the  proposal  demonstrate  understanding  of  the  feasibility  and 
ability  to  successfully  perform  the  task  in  the  following  responses: 

Rating* 

O  B  A  M  U 


1 .  The  proposal  provides  a  description  of  the 
possible  areas  to  be  investigated  in  researching 
the  task. 

2 .  The  proposal  provides  a  detailed  description 
of  the  technical  approach  including  a  detailed 
step-by-step  procedure  and  methodology 
which  would  be  used  in  accomplishing  the  task. 

3 .  The  proposal  identifies  additional  information 
that  would  be  required  to  perform  the  task. 

4.  The  proposal  contains  a  detailed  work  plan  for 
implementation. 

5.  The  proposal  contains  a  product  outline  describing 
what  would  be  the  expected  deliveraUes  and  results 
from  this  task. 

6.  The  proposal  contains  manhouts  by  labor  category. 

7.  Overall  rating. 

•Ratings-  0=  Outstanding,  B=  Better,  A=  Average,  M=  Marginal,  U=  Unsatisfactory 
Narrate  any  strengths  and  weaknesses  belo^^ : 
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The  simpler  format  presented  abo\  e  allows  the  TEC  to  reduce  the  time  required 
in  a  technical  evaluation.  It  also  clarifies  for  the  evaluator  what  the  requiring  activity  views 
as  important  in  an  understandable  format.  The  smaller  number  of  considerations  also 
reduces  the  probability  of  subfactor  dilution.  Note  that  all  considerations  receive  an 
adjectival  rating  with  an  overall  rating  included  at  the  bottom  of  the  e\  aluation  sheet.  Each 
consideration  is  not  weighted  but  reviewed  in  lieu  of  all  other  considerations.  In  the  prior 
example  there  would  have  been  27  ratings,  one  for  each  consideration.  This  simpler  format 
permits  the  evaluator  to  use  judgment  during  the  evaluation  by  only  requiring  the  evaluator 
to  review  six  considerations  before  making  a  determination  on  the  overall  rating. 

t 

c.  Example  #  3 

The  final  example  was  obser\'ed  during  a  meeting  between  a  requiring  activity 
and  contracting  personnel  when  discussing  the  use  of  "best  value"  to  procure  a  series  of 
environmental  tests.  The  following  technical  evaluation  criteria  were  submitted  to  the 
contracting  activity  for  approval  prior  to  the  solicitation. 

TECHNICAL  EVALUATION  CRITERIA 

Factor  Weightings 


1 .  Technical  75 

a.  Environmental  and  safety  tests  |5er  SOW  25 

1)  28  day  temp  and  humidity  5 

2)  transportation  vibration  5 

3)  fast  cookoff  5 

4)  slow  cookoff  5 

5)  bullet  impact  5 

b.  Qualified  personnel  per  SOW  20 

1)  senior  test  engineer  5 

2)  test  engineers  5 

3)  test  technicians  5 

4)  security  staff  5 

c.  Adequacy  of  facilities  20 

1)  proper  test  equipment  10 

2)  adequate  security  and  storage  5 

3)  handle  test  items  safety  5 
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d.  Performance  of  similar  tasks  10 

1)  depth  of  understanding  2 

2)  re^istic  techniques  2 

3)  technical  approach  2 

4)  experience  2 

5)  data  requirements  2 

The  problem  with  the  abo\e  criteria  is  the  fact  that  they  are  based  more  on 
responsibility  type  issues  than  on  issues  related  to  varying  degrees  of  quality  or  technical 
merit.  In  essence,  the  criteria  focused  more  on  whether  an  offeror,  based  on  his/her 
understanding  of  the  tasks  involved  and  the  facilities  required  could  pjerform  the  tests 
required  in  the  SOW.  Most  of  the  criteria  could  be  addressed  with  a  "yes"  or  "no" 
response,  i.e.,  either  the  offeror  has  the  ability  to  fulfill  the  SOW  or  he  does  not.  The 
meeting  determined  that  there  was  little  differentiation  based  on  the  accuracy,  timeliness  or 
quality  of  the  tests  required  by  the  SOW. 

After  approximately  one  hour  of  honest  and  upfront  dialogue  by  both  parties,  it 
was  decided  that  a  low  cost  technically  acceptable  approach  for  the  acquisition  was  optimal. 
It  is  this  researcher's  opinion  that  a  key  in  the  decision  was  the  contract  spjecialist's 
understanding  of  the  technical  requirements.  Without  this  understanding  and  the  ability  to 
ask  relevant  questions  of  the  requiring  activity,  a  "best  value”  strategy  may  have  been 
pursued  that  could  have  resulted  in  the  payment  of  a  premium  when  a  premium  was  not 
necessary. 

3.  Causes  of  Problems  in  the  Identification  of  Evaluation  Criteria 

Reasons  provided  by  interviewees  for  the  causes  of  problems  with  evaluation 
criteria  are;  1)  the  nature  of  the  PWS,  2)  a  lack  of  time  to  p)lan  and  construct  the  criteria,  3) 
a  lack  of  training  for  both  technical  and  contracting  personnel,  4)  a  lack  of  communication 
between  Government  and  industry  and,  5)  a  lack  of  feedback  conducted  after  the  evaluation 
process. 
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a.  Nature  of  the  Requirement 

A  vague  PWS  is  a  major  reason  offered  for  difficulties  in  establishing 
evaluation  criteria.  If  the  PWS  was  well  defined  it  would  be  easy  to  understand  w  hat  is 
important  in  the  performance  of  the  tasks  since  the  desired  outcome  is  know  n.  In  the  case 
of  a  vague  PWS,  the  desired  outcome  is  not  so  certain.  Due  to  the  dynamic  nature  of  many 
programs,  the  requiring  activity  may  have  little  idea  of  the  tasks  that  will  be  required  tw  o 
years  in  the  future.  Therefore,  it  is  difficult  to  select  evaluation  criteria  that  will  lead  to  the 
most  ad^'antageous  procurement  decision. 

b.  Lack  of  Time 

Respondents  to  the  suiAcy  who  mentioned  evaluation  criteria  as  a  problem 
generally  believe  that  the  requiring  acti\’ity  has  the  experience  to  select  and  tailor  the  correct 
evaluation  criteria.  However,  the  requiring  activity  does  not  always  take  the  time  to 
understand  and  really  get  at  the  heart  of  what  is  important  in  the  performance  of  the  contract 
and  what  criteria  w  ill  really  differentiate  between  offerors.  Typically,  evaluation  criteria  are 
selected  at  the  requiring  activity  by  the  technical  code  requiring  the  serv  ices  and  forwarded 
to  the  contracting  activity  or  department  for  approval.  To  save  time  in  developing  selection 
criteria,  the  requiring  activity  normally  reviews  the  evaluation  criteria  for  past  and  present 
contracts  and  utilizes  the  same  criteria  for  the  current  requirement. 

c .  Training  Deficiencies 

The  lack  of  appropriate  training  for  both  technical  and  contracting  personnel  is 
another  reason  why  poor  evaluation  criteria  are  written.  Technical  personnel  are  not  always 
familiar  with  the  entire  source  selection  process  and  may  not  understand  why  evaluation 
criteria  are  so  crucial  in  the  proposal  evaluation  phase.  This  is  especially  true  in  the  area  d" 
engineering  and  technical  services  where  the  contract  period  of  performance  may  extend  up 
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to  five  years  if  all  options  are  exercised.  Due  to  this  considerable  length  of  time  between 
evaluations,  \  er\’  few  trained  personnel  from  the  pre\  ious  selection  process  are  available  to 
relate  the  lessons  learned  from  their  pr^'vious  experience. 

Technical  personnel  are  also  not  always  familiar  with  the  different  source 
selection  methods  used  to  procure  services  and  how  these  methods  relate  to  the 
development  of  evaluation  criteria.  For  example,  technically  acceptable  low  offeror  source 
selections  would  emphasize  responsibility  criteria  w'hereas  "best  value"  source  selections 
emphasize  the  qualitative  criteria.  Contracting  personnel,  familiar  with  different  methods  of 
source  selection,  generally  lack  the  technical  training  required  to  understand  what  the 
requinng  acti\  ity  needs  and  therefore  ha\e  difficulties  in  assisting  the  requiring  activity 
develop  the  proper  evaluation  criteria. 

Training  in  the  selection  and  design  of  evaluation  criteria  is  normally  provided 
by  the  contracting  organization  in  the  form  of  written  guidance.  Most  of  the  contracting 
activities  visited  ha\e  provided  detailed  instructions  on  the  different  source  selection 
methods  w'ith  generic  examples  of  criteria  for  the  requiring  activity  to  utilize. 

d.  Communications  among  the  Acquisition  Planning  Team 

Communications  between  members  of  the  eicquisition  planning  team  consisting 
of  technical  and  contracts  p)ersonnel  is  critical,  but  not  always  sound.  Technical  personnel 
require  an  understanding  of  acquisition  procedures  and  contracting  personnel  require  an 
understanding  of  the  technical  aspiects  of  the  PWS.  To  assist  in  the  communication  flow, 
centralized  contracting  activities  like  NRCCs  sometimes  employ  Requiring  Activity 
Contracting  Assistants  (RACAs)  who  assist  the  requiring  activity  in  the  construction  of  a 
procurement  package  for  NRCC  procurement.  Activities  with  technical  and  contracting 
departments  at  one  location,  have  the  advantage  of  proximity  however,  from  this 
researcher's  experience  both  centralized  and  decentralized  organizations  are  susceptible  to 
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communicaiion  breakdowns  between  technical  and  acquisition  personnel.  This 
communication  breakdow  n  is  normally  attributed  to  the  personal  relationships  between 
technical  and  acquisition  personnel  w  here  both  have  preconcei^'ed  notions  about  the  other. 
For  instance,  some  technical  personnel  maN’  view  contracting  personnel  as  "clerks",  always 
looking  for  the  lowest  price.  Some  contracting  personnel  may  believe  that  technical 
personnel  desire  to  circumvent  the  system  whene\er  possible.  The  communication 
breakdow  n  presents  itself  in  the  less  than  optimal  design  of  evaluation  criteria  for  "best 
value"  evaluations. 

e.  Communications  with  Potential  Offerors 

Early  communication  with  potential  offerors  is  a  method  of  recei\ing  assistance 
in  the  preparation  of  evaluation  criteria.  Draft  RFPs  and  presolicitation  conferences  are 
rarely  used  for  engineering  and  technical  ser\'ices  contracts  because  most  procurement 
personnel  believe  that  the  process  is  not  necessary  and/or  is  too  time  consuming.  The 
majority  of  personnel  inteiA'iewed  were  of  the  opinion  that  draft  RFPs  and  presolicitation 
conferences  were  worthw  hile  for  a  new  procurement  of  complex  ser\’ices  but  not  worth  the 
effort  for  most  engineering  and  technical  sendees  contracts  even  though  "best  value" 
evaluations  w’ere  required.  The  following  reasons  support  this  opinion:  1)  not  all 
contractors  attend  presolicitation  conferences  or  comment  on  draft  RFPs,  2)  regular  RFPs 
elicit  the  same  questions  as  draft  RFPs  and,  3)  contractors  already  have  a  good 
understanding  of  engineering  and  technical  seiA'ice  RFPs  and  the  evaluation  criteria  utilized. 
When  considering  the  years  of  experience  most  contractors  have  with  Government 
contracts  and  the  emphasis  on  the  use  standardized  RFPs  at  most  activities,  the  assertions 
for  not  using  draft  RFPs  and  presolicitation  conferences  for  all  "best  value"  procurements 
are  plausible. 
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Proponents  for  the  use  of  draft  RFPs  and  presolicitation  conferences  for  all 
"best  value"  procurements  counter  that  not  only  do  these  tools  assist  in  improving  the 
quality  of  RFPs,  but  in  the  long  run,  improve  the  relationship  between  Go\  ernment  and 
industry.  Regardless  of  the  implications  or  the  need,  it  was  observ  ed  during  this  research 
that  draft  RFPs  and  presolicitation  conferences  were  rarely  used  in  "best  value" 
procurements  of  engineering  and  technical  sen  ices. 

/.  Lack  of  Feedback 

One  final  cause  attributed  to  improperly  developed  evaluation  criteria,  was  a 
lack  of  feedback  or  lessons  learned  at  the  requiring  activity  after  the  evaluation. 
Specifically,  very  few  interviewees  admitted  to  seriously  meeting  after  the  evaluation  to 
discuss  the  strong  points  or  problems  with  the  evaluation  criteria.  Man\’  felt  that  a  feedback 
session  was  a  poor  use  of  time  since  most  of  the  members  involved  in  the  technical 
evaluation  are  tired  of  the  process  and  very  few  of  the  personnel  who  participated  in  this 
evaluation  w  ill  be  present  for  the  next  one.  The  result,  is  an  investment  of  time  with  limited 
learning  for  future  procurements. 

5 .  Summary 

Identifying  the  projjer  evaluation  criteria  is  crucial  to  the  success  of  "best  value" 
evaluations.  However,  according  to  the  survey  respondents,  selecting  the  evaluation 
criteria  which  will  differentiate  between  offerors  and  eventually  lead  to  the  most 
advantageous  procurement  decision  is  a  difficult  task  w'hich  is  not  always  successfully 
completed.  Reasons  for  the  problems  in  identifying  adequate  criteria  occur  due  to  the:  1) 
the  nature  of  the  PWS,  2)  time  for  planning,  3)  training  for  both  contracting  and  technical 
personnel  and  4)  a  failure  in  communication  between  contracting  and  technical  personnel. 
In  addition,  draft  RPTs  and  feedback  sessions  w'ere  noted  as  methods  of  assisting  in  the 
construction  of  criteria,  both  of  w-hich  are  rarely  utilized. 
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C.  PROBLEM  #2:  THE  TECHNICAL  EVALUATION 

Technical  evaluation  personnel  are  not  always  educated  concerning  the  importance  of 
the  technical  evaluation  to  the  "best  \'alue’'  source  selection  process. 

1 .  The  Importance  of  Technical  Evaluations 

The  technical  evaluation  process  is  critical  in  determining  which  proposal  offers  the 
most  advantageous  procurement  decision  to  the  Government.  An  undisciplined  technical 
^valuation  can  lead  to  the  selection  of  an  offeror  who  does  not  of  the  "best  value",  or  to 
the  loss  of  a  protest  questioning  the  integrity  of  the  source  selection  process.  A  strong 
technical  evaluation  is;  1)  required  to  assist  the  SSA  in  deciding  contract  award,  2) 
necessary  for  discussions  and  debrief  and,  3)  a  defense  against  protest.  Problems  with 
technical  evaluations  of  proposals  were  mentioned  by  nearly  half  of  the  Government 
personnel  inter\’iewed.  Many  believe  technical  evaluations  were  a  serious  problem,  "Our 
biggest  problem  with  the  greatest  value  method  is  the  technical  evaluators  who  want  to 
quantify  everything  in  numbers  and  not  provide  narrative".  Others  thought  that  problems 
with  technical  evaluations  were  relatively  minor,  "Correcting  technical  narrative  is  nothing 
more  than  a  routine  procedure  similar  to  submitting  correspondence  through  a  supervisor 
prior  to  release". 

2.  Deficiencies  Mentioned  by  Contracting  and  Technical  Personnel 

Regarding  the  Technical  Evaluation  Process 

Several  comments  collected  during  this  research  regarding  the  problems  with  the 
technical  evaluation  process  are  presented  below. 

Technical  evaluators  grade  too  high.  They  rarely  give  failing  grades. 

Being  a  member  of  a  technical  evaluation  board  is  not  looked  upon  highly  by  the 
senior  engineers.  The  assignment  to  the  technical  panel  needs  to  be  recognized  as  an 
activity  with  value.  Also,  technical  people  see  contracting  personnel  as  clerks  which 
does  even  less  to  help  the  process. 
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Technical  ex  aluators  are  not  well  trained.  They  usually  do  it  once  in  their  career  and 
do  not  want  to  take  the  responsibility.  Contracting  personnel  can  help  by  making  the 
process  (technical  evaluation)  as  painless  as  possible. 

.  Technical  evaluation  committees  are  made  up  of  technicians  and  engineers  who  are 
not  sensitive  to  the  (source  selection)  process.  Strong  preferences  exist  at  each  activity. 

Technical  evaluations  are  bad-  everyw'here!  The  contracting  officer  needs  to  ensure 
the  technical  evaluators  are  trained  to  properly  cany  out  their  duties. 

Heads  of  requiring  activities  blame  contracting  officers  because  they  are  told  how  to 
conduct  technical  evaluations  and  do  not  alw’ays  believe  they  get  the  best  possible 
contractor  since  they  feel  constrained  by  the  rules. 

Like  all  other  activities  I  know’  of,  we  too  have  a  problem  with  technical  narratix  e. 

There  are  normally  lots  of  problems  with  the  technical  evaluation  process.  It  is 
alw'ays  an  uphill  battle.  The  technical  evaluation  panel  is  usually  the  8th  priority  for  any 
engineer.  It  is  also  very  painful  to  get  narrative  comments  from  engineers;  things  that 
will  live  up  to  legal  scrutiny  that  is. 

The  technical  evaluation  committee  did  not  work  well  together  as  a  team.  It  took  six 
months  longer  than  scheduled  to  award  the  contract  because  of  this.  The  narrative  had 
to  go  back  several  times  for  rewrite. 

Technical  evaluators  see  things  in  black  and  white.  They  like  to  use  numbers  when 
most  of  the  differences  in  proposals  cannot  be  equated  to  a  numerical  difference. 

The  most  common  complaint  concerning  the  technical  evaluation  was  the  lack  of 
technical  narrative  which  is  required  to  document  a  "best  value"  award  decision.  As 
alluded  to  in  the  comments  abo\e,  it  appears  that  technical  evaluators  prefer  the  use  of 
numerical  grades  o\’er  narrative  to  document  the  evaluation.  This  researcher  has  seen  four 
examples  at  two  activities  of  contract  awards  delayed  for  up  to  six  months  primarily  due  to 
submissions  of  poor  narratives  by  the  technical  evaluation  committees  (TEC).  The 
common  scenario  has  the  TEC  submitting  an  evaluation  to  the  contracting  officer  who  feels 
that  it  is  inadequate  to  support  an  award.  The  evaluation  is  sent  back  to  the  TEC,  redone, 
and  submitted  back  to  the  contracting  officer.  This  may  occur  three  or  four  times  before  a 
contract  award  is  made.  In  one  example  involving  two  offerors  for  a  five  year  $7  million 
services  contract,  the  technical  evaluation  was  sent  back  to  the  TEC  five  times  between 
February  and  July  1988.  In  between  were  two  meetings  and  several  conversations 
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between  the  contract  specialist  and  the  requiring  department.  According  to  the  contract 
specialist,  the  procurement  did  attain  the  most  advantageous  procurement  decision 
however,  it  was  more  costly  than  it  should  have  been  in  terms  of  time  and  resources. 

In  an  extreme  example,  an  offeror  in  a  competition  for  engineering  suppxDrt 
ser\  ices  proposed  to  perform  a  PWS  under  a  cost  sharing  arrangement  with  no  fee  in  lieu 
of  a  CPFF  contract.  The  contracting  officer  reasoned  that  the  firm  was  trying  to  remain  in 
business.  This  offeror  proposed  such  a  low  price  and  favorable  terms  that  on  overall 
points  the  proposal  was  a  clear  winner.  The  second  offeror,  the  incumbent,  out  scored  the 
low'  priced  offeror  on  technical  points  52  to  43  although  both  were  technically  acceptable. 
The  contracting  officer  admitted  that  he  realized  the  requiring  activity  desired  the  incumbent 
offeror  because  of  the  higher  technical  score.  Unfortunately,  he  could  not,  even  after 
several  meetings,  extract  the  technical  narrative  necessary'  to  make  award  to  the  incumbent 
offeror.  As  a  result,  the  contracting  officer  had  no  justification  but  to  award  to  the  low 
priced  technically  acceptable  offeror.  One  year  into  the  contract,  the  requiring  activity  is  not 
pleased  with  the  contractor's  performance  and  maj'  elect  not  to  exercise  the  next  option  and 
recompete  the  PWS.  Although  extreme,  the  above  example  illustrates  what  could  happen 
W'ithout  adequate  narrative. 

3 .  Systems  to  Prevent  Problems  with  the  Technical  Evaluation 

All  of  the  activities  interviewed  have  in  place  instructions  and  recommended 
procedures  to  train  technical  evaluators  and  facilitate  the  technical  evaluation.  NRCC, 
Philadelphia,  has  a  four  page  instruction  on  the  evaluation  of  offeror's  technical  proposals 
[Ref.  16].  Included  w'ith  the  instruction  is  a  note  from  the  Commanding  Officer  to  the  TEC 
which  lists  and  describes  common  defects  in  technical  evaluations.  NRCC,  Washington, 
D.C.,  in  a  package  for  all  customer  activities,  not  only  includes  instructions  to  the  TEC,  but 
also  includes  sample  narratives  justifying  an  evaluator's  rating  on  an  evaluation  factor. 
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Naval  Supply  Center  (NSC),  Puget  Sound,  requires  technical  e\  aluators  to  pass  a  test  on 
the  source  selection  e\  aluation  process  prior  to  assignment  to  the  TEC.  Included  in  the  test 
are  21  questions;  some  requiring  explanations,  while  others  are  true  and  false.  The  test  is 
presented  in  Appendix  E. 

In  addition  to  written  instructions,  technical  evaluators  are  further  instructed 
verbally  prior  to  commencement  of  the  actual  technical  evaluation.  This  instruction  is 
delivered  by  either  the  contracting  officer,  his/her  representati\’e  or  the  chairperson  of  the 
TEC.  For  centralized  contracting  activities  like  NRCCs,  technical  evaluators  are 
encouraged  to  utilize  conference  space  at  the  contracting  activity  to  conduct  their  technical 
evaluations.  This  improves  the  process  by  sequestering  the  technical  evaluators  away  from 
their  day  to  day  job  requirements  while  dedicating  their  complete  attention  to  the  evaluation 
of  technical  proposals.  It  also  places  the  TEC  where  they  can  receive  immediate  feedback 
from  the  contracting  officer  and  legal  counsel 

4 .  Causes  of  Problems  in  Technical  Evaluations 

With  all  of  these  systems  in  place,  why  do  problems  continue  to  exist  betw  een  the 
contracting  and  technical  personnel  regarding  the  technical  evaluation?  The  major  reason  is 
due  to  a  lack  of  management  commitment  to  the  process.  Management,  on  both  sides,  does 
not  always  comm  t  the  resources  considered  necessary  to  ensure  that  an  accurate  and 
expeditious  technical  evaluation  is  completed.  This  can  be  attributed  to  a  lack  of  personnel 
resources,  time  and  an  understanding  of  the  importance  of  a  disciplined  evaluation.  While 
discussing  the  importance  of  a  disciplined  evaluation,  one  contract  specialist  admitted  that 
she  prefers  to  have  a  protest  on  occasion,  if  only  to  have  the  management  personnel  at  the 
technical  activity  understand  the  need  for  a  disciplined  technical  evaluation  process. 

One  of  the  most  frequently  mentioned  constraints  on  the  technical  evaluation 
process  was  that  the  engineers  employed  on  technical  evaluations  have  full  time  jobs  in 
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addition  to  the  assignment  on  the  technical  panel.  As  a  result,  thcj'  can  only  dedicate 
random  portions  of  their  time  to  the  evaluation.  In  addition,  there  exists  a  lack  of  incenti\  e 
to  become  a  member  of  the  evaluation  team.  Normally,  individuals  are  not  evaluated  nor 
do  they  receive  performance  awards  based  on  their  participation  on  a  TEC.  Therefore, 
there  exists  little  motivation  to  expend  great  effort  during  the  evaluation.  The  obvious 
motivation  is  in  choosing  the  best  contractor  the  requiring  activity  will  have  to  work  with 
for  the  next  four  or  five  years.  Unfortunately,  this  is  not  always  a  superior  motivating 
factor.  In  many  instances,  junior  and  new  personnel  are  assigned  to  the  TEC  who  have 
little  understanding  of  the  process. 

Contracting  organizations  can  assist  in  the  technical  evaluation  in  a  variety  of 
ways.  First,  contracting  officers  and  specialists  can  become  familiar  with  the  PWS  to  the 
point  where  they  understand  uhat  the  differences  in  technical  proposals  mean  to  successful 
completion  of  the  PWS.  This  would  assist  the  technical  evaluation  panel  in  verbalizing  the 
differences  in  the  technical  narratixe.  Second,  the  contracting  organization  can  be  available 
and  accessible  at  all  times  to  answer  questions  from  the  TEC.  Finally,  contracting 
organizations  can  offer  innovative  training  and  feedback  mechanisms  like  the  test  at  NSC 
Puget  Sound  to  improve  technical  evaluation  procedures. 

5 .  Summary 

Technical  evaluations  are  an  important  element  of  "best  value"  source  selections. 
Without  a  disciplined  technical  evaluation  and  adequate  documentation,  it  is  difficult  if  not 
impossible  to  make  the  most  advantageous  procurement  decision.  Both  contracting  and 
technical  personnel  have  issues  concerning  the  technical  evaluation  even  though  systems 
are  in  place  to  alleviate  these  issues.  A  lack  of  management  commitment  and  understanding 
were  the  most  frequently  cited  causes  of  problems  with  technical  evaluations  for 
engineering  and  technical  services  contracts. 
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D.  PROBLEM  #3:  CONTRACTOR  GAMING  ON  COST  AND  TECHNICAL 

PROPOSALS 

1 .  Gaining  Defined 

The  term  "gaming"  as  used  in  this  thesis,  is  defined  as  an  inno\’ation  in  cost  or 
technical  approach  proposed  by  an  offeror  without  adequate  justification.  These 
innovations  are  used  to  improve  technical  proposals  and/or  lower  costs.  Under  gaming, 
the  contractor  hcis  little  or  no  intention  of  fulfilling  the  innovations  proposed.  Three 
reasons  offered  for  gaming  include;  1)  the  hope  of  becoming  more  competitive  for  award, 
2)  a  feeling  of  distrust  in  a  system  advertised  to  focus  on  technical  over  cost,  and  3)  to 
protect  the  firm  from  the  Government,  which  is  "notorious  for  being  tough  on  customers". 
This  final  view  considers  gaming  as  necessary  in  order  to  add  a  cushion  to  ov  erhead  costs 
to  remain  in  business  or  generate  an  acceptable  fee. 

2.  Responses  From  Interviews  on  the  Gaming  Problem 

Several  comments  collected  during  this  research  which  is  indicative  of  the  gaming 
problem  are  presented  below . 

We  get  it  (gaming)  all  the  time.  Typically  we  know  where  the  problems  exist  and 
look  at  those  areas.  Gaming  is  used  most  often  in  overhead  and  skill  mix.  You  also 
see  a  lot  of  gaming  in  other  direct  costs  and  escalation  rates  when  they  are  not 
normalized. 

I  do  not  believe  w  e  get  the  hours  proposed  in  uncompensated  overtime. 

Some  contractors  come  in  with  great  costs.  However,  when  investigated  more 
closely  you  note  the  costs  do  not  match  the  technical  approach. 

The  problem  we  get  with  engineering  support  services  is  a  contractor  bidding  the 
"A"  team  and  you  w  ind  up  getting  the  "C"  team. 

"Best  value"  allows  the  technical  people  to  use  judgment.  Personnel  experience  is 
usually  the  most  important  factor  since  sample  tasks  can  be  gamed  by  a  good  technical 
writer. 

Many  contractors  submit  reliable  cost  estimates;  others  do  not.  Cost  realism  to 
control  gaming  costs  us  lots  of  extra  work,  time  and  effort  because  you  cannot  rely  on 
some  contractors'  data. 
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Contractors  game  the  GoNernment  on  both  technical  and  cost.  You  should  be 
suspicious  w  hen  costs  submitted  b\'  off  erors  are  far  apart.  Contractors  u  ill  tell  >ou  one 
thing  in  the  technical  proposal  and  the  costs  do  not  match  up. 

There  is  a  gaming  problem  with  costs.  Companies  form  joint  ventures  w  ith  other 
companies  and  nobody  with  resumes  provided  is  assigned  to  the  company  being 
evaluated.  Many  companies  setup  new  cost  centers  to  bid  on  a  proposal. 

Lots  of  companies  play  games  u  ith  cost  data  after  the  contract  is  aw  arded. 

3 .  Where  is  Gaming  Found? 

According  to  information  obtained  during  this  research,  gaming  is  found  in  both 
cost  and  technical  proposals.  In  the  cost  area,  frequently  mentioned  gamed  costs  include 
direct  labor  rates  and  o\  erhead  proposed  b\  the  prime  and  subcontractors.  Escalation  rates 
and  other  direct  charges  are  also  gamed  when  not  normalized.  Areas  mentioned  where 
technical  gaming  occurs  included;  1)  changes  in  the  proposed  labor  skill  mix,  2)  bait  and 
switch  tactics  where  the  key  personnel  proposed  are  not  delivered,  and  3)  technical 
approaches  defined  in  sample  tasks  which  are  unworkable  from  the  cost  standpoint.  In 
fact,  an  example  of  contractor  gaming  was  related  to  this  researcher  in  ex  ery  factor  and 
subfactor  used  in  the  evaluation  of  engineering  and  technical  ser%  ices  contracts. 

After  award,  several  inter\  iewees  mentioned  task  proposals  from  contractors  w  ith 
skill  mixes  which  do  not  agree  with  the  overall  proposal  skill  mix  awarded  the  contract. 
For  instance,  after  award,  a  contractor  may  game  by  proposing  a  larger  percentage  of 
higher  grade  personnel  than  indicated  in  the  original  labor  mix.  The  result  is  more 
overhead  for  the  contractor  and  a  higher  rate  of  fund  expenditure  by  the  Government.  This 
problem  could  be  partly  attributable  to  a  poor  estimate  of  the  labor  skill  mix  prepared  by 
Government  personnel  or  the  inability  of  the  COTR  or  ordering  officer  to  negotiate  a 
correct  skill  mix  on  the  delivery  or  task  order.  For  this  reason,  many  consider  cost 
overruns  as  an  "inherent"  problem  in  serx  ice  contracting. 
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4.  Examples  of  Contractor  Gaming 


The  following  are  examples  of  suspected  contractor  gaming  found  in  the  cost 
evaluation  sections  of  two  post  business  clearance  memorandums  examined  during  this 
research.  Both  clearances  are  for  engineering  and  technical  seiA'ices  and  propxjse  one  base 
year  with  four  option  years.  The  e\  aluated  costs  for  one  of  the  contracts  is  approximately 
$16  million.  The  second  contract  has  evaluated  costs  of  approximately  $30  million. 

In  the  first  instance  of  suspected  gaming,  one  of  the  contractors  has  submitted 
labor  categories  with  a  wide  range  of  hourly  rates.  Shown  below  is  a  chart  which  reveals 
how  the  contractor  distributed  the  hours  within  a  given  labor  category  among  the  various 
classifications  that  make  up  the  labor  category'.  The  direct  hourly  rate  is  also  shown  for 
each  classification  and  the  percentage  of  effort  for  each  classification  is  shown  in  the 
parenthesis.  This  example  only  includes  two  of  the  nine  labor  categories  for  illustration. 


E18 

250 

Proeram  Manager 
hrs  (104) 

$33.38 

E8 

2200 

hrs  (90%) 

16.46 

$18.19 

EL 

250 

Equipment  Specialist 
hrs 

$12.51 

EJ 

250 

hrs 

10.82 

ED 

250 

hrs 

7.35 

EC 

8250 

hrs  (92%) 

6.85 

$7.13 

As  demonstrated  above,  and  in  six  of  the  seven  remaining  labor  categories,  the  majority  of 
all  hours  proposed  are  at  the  low  end  of  the  pay  scale.  This  in  itself  is  not  an  indication  of 
contractor  gaming.  However,  a  comparison  to  the  composite  rates  of  other  offerors  and  the 
Government's  estimation  based  on  previous  efforts  indicates  that  the  rates  were  not 
accurate,  i.e.,  the  above  rates  are  considerably  lower.  In  addition,  the  comparison  of 
hourly  rates  with  the  annual  salaries  displayed  in  another  section  of  the  proposal  reflects 
wide  disparities.  "As  an  example,  the  gentleman  proposed  as  the  program  manger  has  an 
annual  salary  of  $83,  512  whereas  the  annual  salaiy  in  the  cost  proposal  is  $37,835". 


In  another  example  in  the  same  evaluation,  one  of  the  contractors  proposed 
overhead  rates  well  below  the  forecasted  rate  for  the  next  three  years.  Whereas  the 
forecasted  rate  averaged  72%,  the  proposed  rate  a\  eraged  59%.  The  justification  offered 
b}'  the  contractor  for  the  discrepancy  was  that  they  w'ere  establishing  a  new  cost  center  for 
the  effort.  The>'  offered  three  reasons  for  the  the  new  cost  center  including: 

1.  Type  of  work  to  be  performed  under  the  contract  was  different  from  existing 
contracts. 

2.  The  current  division's  rates  would  include  costs  the  company  feels  should  not  be 
allocated  to  the  contract. 

3 .  Existing  division's  rates  are  not  comp)etitive  for  this  procurement. 

DCAA  felt  that  the  above  reasons  did  have  merit,  but  were  not  sufficient  for  a 
number  of  technical  and  cost  reasons.  Since  there  was  no  cost  information  for  the  new 
proposed  cost  center,  the  Government  utilized  historic  data  and  the  original  forecast  to 
develop  rates  which  turned  out  to  be  considerably  higher  than  originally  proposed  by  the 
contractor. 

In  the  final  example,  evaluated  direct  labor  costs  proposed  by  a  subcontractor  were 
increased  by  over  80%  as  a  result  of  a  cost  realism  analysis.  The  reason  for  the  increase  in 
evaluated  costs  is  the  disparity  bet\^•een  resume  salaries  presented  in  an  appendix  to  the  cost 
proposal  and  the  hourly  rates  presented  in  the  cost  proposal.  Once  again  the  proposal 
average  is  w  ell  below  other  contractors'  proposals  and  the  Government  cost  estimate  based 
on  previous  experience.  Table  3  was  taken  from  the  cost  evaluation: 


TABLE  3 


Category 

Resume  Average 

ProDosal  Average 

Sr.  Lev.  Engr.  Mgr. 

$32.90 

$22.10 

Sr.  Lev.  Engr.  Desgn 

34.01 

17.04 

Sr.  Lev.  Engr.  Rel. 

20.05 

17.04 

Jmy.  Engr. 

17.63 

10.36 

Engr.  Desgn 

25.13 

16.92 

Engr.  Rel. 

25.27 

16.92 

Computer  Scientist 

30.85 

12.69 
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As  evidenced  in  Tabic  3,  a  large  disparity  exists  between  the  resume  rates  and  the  proposed 
rates.  The  e\  aluated  cost  was  re\  ised  upward  as  a  result. 

5 .  Uncompensated  Overtime 

Uncompensated  0\  ertime  (UCOT)  was  mentioned  as  a  gaming  problem  by  a  large 
number  of  Government  and  industry  personnel.  UCOT  is  defined  in  a  DEARS  change 
published  in  the  Federal  Register,  Vol.  56,  No.  172,  page  43987,  on  September  5,  1991  as 
follows: 

The  hours  worked  in  excess  of  the  normal  8  hours  per  day,  40  hours  per  week  by 
professional  employees  who  are  exempt  from  the  Fair  Labor  Standards  Act  without 
additional  compensation.  Compensated  personnel  absences,  such  as  holidays, 
vacations,  and  sick  leave,  shall  be  included  in  the  normal  work  week  for  purposes  of 
computing  uncompensated  overtime  hours. 

(The)  "Uncompensated  Overtime"  rate  is  the  rate  which  results  from  multiplying 
the  hourly  rate  from  a  40  hour  work  week  by  40,  and  then  dividing  by  the  proposed 
hours  per  week.  For  example,  45  hours  proposed  on  a  40-hour  work  week  basis  at 
$20.00  would  be  converted  to  an  uncompensated  overtime  rate  of  $17.78  per  hour. 
($20X40)  divided  by  45=$  17.78. 

The  DEARS  change  requires  contracting  officers  to  evaluate  UCOT  for  technical 
risk  in  fulfilling  contract  requirements.  Of  particular  note  arc: 

1 .  Unrealistically  low  labor  rates  or  other  costs  that  may  result  in  quality  or  service 
shortfalls;  and 

2.  Unbalanced  distribution  of  uncompensated  overtime  among  skill  levels  and  its  use 
in  key  technical  positions. 

This  policy  of  evaluating  the  technical  risk  of  UCOT  was  tested  in  Comptroller  General 
decision  B-242020  of  March  21,  1991  in  the  matter  of  Quantum  Research,  Inc.  (QRI).  A 
short  summary  of  the  decision  follow  s: 

QRI  bid  on  a  Government  contract  in  which  it  indicated  that  its  professional 
employees  work  2600  hours  per  year  or  50  hours  per  week.  (One  industry  standard  is 
20^  hours  annually,  a  40  hour  week.)  The  contracting  agency  downgraded  QRI's 
proposal  in  part,  to  the  use  of  such  hours.  GAO  concluded  that  the  downgrading  was 
justified,  because  the  2600  hour  work  year  created  a  risk  regarding  the  stability  of 
QRI's  workforce.  [Ref.  17:p.  8] 

Industry's  position  on  UCOT  is  that  its  use  has  placed  undue  emphasis  on  price 
rather  than  quality  in  the  source  selection  process  for  engineering  and  technical  services. 
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Industry  believes  the  the  Government  should  "level  the  playing  field"  by  evaluating 
proposals  for  engineering  and  technical  senices  on  a  "normalized"  40  hour  per  w  eek  basis 
regardless  of  the  work  week  proposed  or  the  acceptability  of  the  accounting  systems  used 
by  offerors.  [Ref.  18;p.  3] 

While  not  endorsing  the  use  of  UCOT,  the  Government  believes  that  disallowing 
its  use  "would  be  detrimental  to  the  integrity  of  the  Government's  acquisition  system  to 
prohibit,  restrict,  or  discourage  any  form  of  price  competition,  including  the  technique  of 
proposing  uncompensated  o\  ertime."  [Ref.  18:p.  3] 

Inier\  iews  indicate  that  Government  acquisition  personnel  recognize  the  problems 
with  excessive  UCOT  and  ha\  e  instituted  controls  to  evaluate  the  increased  technical  risk. 
In  fact,  all  Government  contracting  organizations  visited  during  this  research  claim  to 
evaluate  UCOT  in  light  of  increased  technical  risk.  Several  organizations  review  historic 
UCOT  usage  and  do  not  f)ermit  increasing  UCOT  hours  in  new  proposals.  Even  with 
these  controls  in  place,  this  has  not  ended  complaints  by  many  Government  and  industr>’ 
interviewees  who  claim  they  do  not  believe  the  Government  is  getting  the  UCOT  hours 
proposed.  Moreo\'er,  the  extra  time  and  resources  dedicated  to  evaluating  UCOT  is  not 
worth  the  effort. 

One  senior  Navy  acquisition  official  believes  that  the  problem  with  UCOT  has  "run 
its  course"  in  that  the  controls  instituted  are  sufficient  to  protect  both  Government  and 
industry.  Additionally,  he  believes  the  use  of  excessive  UCOT  is  now’  on  the  decline. 
Judging  from  the  number  of  comments  received  during  this  research  from  both 
Government  and  industry  personnel,  it  is  this  researcher's  belief  that  continued  study  of 
UCOT  should  be  pursued  to  determine  the  costs  and  benefits  of  the  Government's  policy. 
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6 .  Why  Gaming  is  a  Problem 

Gaming  is  a  problem  for  se\  eral  reasons.  First,  gaming  perturbates  the  budgeting 
process  and  can  affect  readiness.  Acti\  ities  budgeting  funds  based  on  a  yearly  proposed 
amount  negotiated  for  an  engineering  and  technical  ser\’ices  contract  are  required  to  curtail, 
cancel  or  obtain  additional  funds  to  continue  services  once  the  budgeted  funds  are 
exhausted.  Secondly,  gaming  increases  the  risks  of  not  selecting  the  most  advantageous 
contractor.  Unwittingly,  a  SSA  may  select  a  contractor  based  on  a  superior  technical 
proposal  that  the  contractor  can  never  deliver.  The  third  reason  gaming  is  a  problem  in 
"best  value"  evaluations  is  the  amount  of  time  and  effort  exp)ended  by  the  SSEB  in  an  effort 
to  combat  it.  All  areas  of  the  cost  and  technical  proposals  require  detailed  screening  and 
comparison  to  avoid  gaming.  Finally,  gaming  increases  the  perception  that  contractors  are 
unethical.  Several  inter\  iewees  mentioned  ethical  concerns  with  gaming,  yet  caveat  their 
comments  with  the  statement  that  the  overwhelming  majority  of  contractors  were  honest 
with  regard  to  their  proposals. 

7 .  Government  EfToiis  to  Combat  Gaming 

The  Government  has  developed  several  methods  to  combat  contractor  gaming  in 
order  to  protect  itself  and  responsible  offerors.  The  first  is  through  use  of  the  cost  realism 
analysis.  As  previously  defined,  the  cost  realism  analysis  is  a  review  of  the  cost  portion  of 
an  offeror's  proposal  to  determine  if  the  costs  are  realistic  for  the  technical  effort  proposed. 
It  is  a  technique  which  should  be  used  where  adequate  price  competition  exists  to  guard 
against  pricing  which  is  unrealistically  low  and  which  may  lead  to  quality  deficiencies, 
performance  shortfalls  and/or  cost  overruns.  The  cost  realism  analysis  allows  the 
contracting  officer  to  gather  and  analyze  the  minimum  data  needed  to  justify  an  offeror's 
proposal  price.  The  cost  realism  analysis  is  an  indicator  of  the  offeror's  understanding  of 
the  RFP's  technical  requirements.  Costs  proposed  by  the  contractor  are  compared  to 
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historical  data  and  the  technical  proposal  and  increased  if  they  are  found  to  be  unrealistic 
pnor  to  making  a  decision  on  award.  Appendix  F  contains  a  matrix  similar  to  one  used  at 
many  activities  as  a  starling  point  for  the  cost  realism  analysis  of  support  serv  ices  contracLs. 
The  matrix  is  a  rough  order  of  magnitude  companson  of  hours  and  costs  between  the 
existing  contract  and  proposals  receiv  ed. 

The  cost  realism  analysis  is  an  effecti\  e  method  used  to  reduce  contractor  gaming. 
Ho«e\er,  the  anahsis  is  often  protested  as  arbitrar\-  and  incorrect.  The  protests  occur 
because  much  of  the  cost  realism  analysis  is  judgmental  in  nature.  Generally  speaking,  the 
Comptroller  General  will  deny  the  protests  if  the  contracting  officer's  analysis  of  the 
proposed  costs  is  sufficient  to  make  a  supportable  judgment  that  the  proposed  costs  are 
realistic.  This  rationale  is  termed  the  "informed  judgment  rule". 

The  cost  realism  analysis  is  not  performed  for  time  and  materials  and  indefinite 
delivery/  indefinite  quantity  contracts  with  fixed  hourly  rates  since  these  rates  will  not  be 
adjusted  upward.  Therefore,  engineering  and  technical  ser\’ices  awarded  under  these 
contract  types  with  fixed  hourly  labor  rates  are  evaluated  under  a  cost  reasonableness 
standard  only  . 

In  addition  to  a  cost  realism  analysis,  another  method  of  combatting  contractor 
gaming  is  by  establishing  cost  ceilings  in  areas  where  the  Go\  emment  suspects  the  use  of 
unrealistic  rales.  Cost  ceilings  tire  negotiated  to  protect  the  Government.  For  instance,  if  a 
contractor  proposes  an  overhead  rale  considered  low  by  the  contracting  officer,  a  ceiling 
can  be  established  requiring  the  contractor  to  remain  below'  the  overhead  rale  ceiling  for  a 
specified  period  of  time.  The  problem  with  imposing  ceilings  is  that  a  ceiling  may  defeat 
the  purpose  of  awarding  the  contract  to  the  most  advantageous  offeror  in  the  first  place. 
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This  is  because,  with  a  ceiling  in  place,  the  contractor  is  more  apt  to  concentrate  on  cost  in 
lieu  of  quality.  According  to  one  interviewee,  imposing  ceilings  for  a  certain  contract  ma\ 
mean  higher  charges  under  a  different  contract. 

Another  method  of  mitigating  gaming  is  through  the  use  of  normalization. 
Normalization  invoh  es  measuring  offerors  against  the  same  cost  standard  or  baseline  of 
circumstances  where  there  are  no  logical  differences  in  approach.  This  technique  is 
typically  applied  in  the  cost  areas  identified  as  other  direct  costs  and  escalation  rates. 
Normalization  has  the  added  benefit  of  reducing  the  time  required  for  cost  realism  analysis 
since  the  elements  normalized  do  not  require  additional  investigation. 

Rejecting  a  proposal  as  non-reponsi\'e  if  costs  are  unrealistic,  or  the  overall 
proposal  is  unbalanced,  is  yet  another  method  of  combatting  gaming.  A  typical  statement, 
taken  from  a  NAVAIR  RFP  to  this  effect  reads: 

The  Government  may  reject  an  offer  as  non-reponsive  if  it  is  materially  unbalanced 
as  to  the  prices  for  the  basic  requirement  and  the  option  quantities.  An  offer  is 
unbalanced  when  it  is  based  on  prices  significantly  less  than  cost  for  some  uork  and 
prices  w  hich  are  significantly  overstated  for  other  work. 

None  of  the  contracting  officers  interviewed  during  onsite  visits  admitted  to  have  eliminated 

offerors  as  non-reponsive  based  on  unrealistic  costs.  In  fact,  many  expressed  concerns 

that  just  such  an  action  is  prohibitix  e  due  to  the  possibiliN  of  protests. 

8 .  Summary 

According  to  interviewxes,  contractor  gaming  is  a  common  practice  when 
submitting  proposals  for  engineering  and  technical  services.  Gaming  in  cost  and  technical 
areas  can  make  an  offeror  more  competitive  for  award.  However,  gaming  can  cause 
perturbations  in  the  budgeting  process  and  require  a  huge  investment  in  time  and  resources 
by  Government  personnel  to  protect  the  integrity  of  the  contracting  system.  The 
Government  employs  a  variety  of  techniques  to  combat  gaming.  Most  are  time  consuming 
but  necessary  to  ensure  that  the  integrity  of  a  "best  value"  evaluation  process  is  kept  intact. 
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E.  PROBLEM  #4:  CONTRACTOR  DISTRUST 


Contractors  do  not  always  trust  "best  value"  e\aluations.  This  claim  is  based  on  the 
perception  that  e\  aluation  criteria  communicated  to  the  offerors  through  the  RFP  are  not 
adequate  to  prepare  a  proposal  fully  responding  to  the  Government's  requirement  and  a 
belief  that  DoD  contracting  officers  continue  to  make  contract  a\va.  ds  based  on  the  low  bid. 

1  .  Comments  Received  Reflecting  a  Lack  of  Trust  in  "Best  Value" 
Evaluations 

We  (Government)  pro\  ide  evers  thing  to  contractors  in  spite  of  their  belief  that  we 
award  on  the  basis  of  low  bid. 

Industry'  is  ver)’  competiti\  e  price  wise.  It  is  hard  to  say  we  do  not  like  a  contractor 
because  his  price  is  loo  low . 

1  received  a  lot  of  high  order  criticism  when  briefing  contractors  about  "best  value" 
last  month.  They  wanted  to  know  the  evaluation  criteria  and  w'eights  we  use  to  rale 
them. 

Everyone  needs  to  change  their  mindset  before  "best  value"  will  happen.  The 
current  mindset  is  low  bid. 

DoD  dollars  are  shrinking  and  customers  still  say  they  want  top  of  the  line. 
Contractors  structure  proposals  in  accordance  with  technical  requirements  because  we 
say  in  the  solicitation  it  is  the  most  important  factor.  Later  we  award  to  the  low  cost 
offeror.  No  w  onder  contractors  don't  trust  the  system. 

Industrv’  feels  we  are  paying  lip  serv  ice  to  the  "best  value"  concept. 

2 .  Systemic  Areas  for  Distrust 

Problem  #4  was  heard  from  15  Government  and  industry  personnel  who  were 
interviewed  during  this  research.  The  views  represented  two  areas  inherent  in  "best  value" 
evaluations  that  generate  contractor  distrust.  Those  areas  are  the  communication  of 
evaluation  criteria  and  a  lack  of  trust  that  the  Government  will  award  contracts  on  a  "best 
value"  basis  in  lieu  of  the  low  bid. 
a.  Evaluation  Criteria 

The  Government  policy  of  communicating  evaluation  criteria  and  their 
respective  weights  to  offerors  can  cause  a  lack  of  trust  between  the  Government  and 
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contractors.  This  is  because  the  evaluation  criteria  used  in  "best  value"  evaluations  to 
assess  the  efficienc}'  and  effecti\  eness  of  an  offeror's  approach  are  ne\  er  completely  or 
objectively  expxjsed  to  offerors  in  the  RFP.  As  a  result,  contractors  claim  they  do  not 
know  v\  hat  they  are  being  e\  aluated  on  and  are  required  to  decide  which  elements  of  the 
proposal  or  PWS  requires  the  most  emphasis.  Upon  not  being  selected  for  contract  award, 
unsuccessful  offerors  attribute  the  cause  to  not  understanding  the  evaluation  criteria.  This 
can  lead  to  a  distrust  of  the  "best  value"  evaluation  process. 

In  a  low  cost  technically  acceptable  competition,  an  offeror  is  well  aware  that  a 
contract  award  will  be  made  solely  on  the  low  cost  among  those  offerors  rated  technically 
acceptable.  In  a  "best  value"  procurement,  the  contract  award  is  based  on  a  combination  of 
factors.  Even  though  section  M  of  an  RFP  may  state  that  the  technical  factor  is  considered 
to  be  more  important  than  cost,  this  does  not  necessarily  mean  the  high  technical  score 
always  wins.  This  fact  presents  a  difficult  dilemma  for  contractors  since  a  decision  on 
where  to  place  their  emphasis,  and  how  much,  could  make  the  difference  between  a 
winning  and  a  losing  proposal.  "Best  value”  source  selection  forces  offerors  to  ask 
themselves,  "What  should  my  proposed  strategy  be,  high  technical,  low  cost  or  something 
in  between?" 

At  the  factor  level,  the  same  contractor  dilemma  exists.  One  Government 
interviewee  stated  that  sometimes  the  TEC  looks  for  "buzzwords"  while  evaluating 
proposals  and  other  times  the  TEC  looks  for  concepts.  He  further  stated  that  evaluation 
criteria  are  generally  held  "close  to  the  chest"  in  that  they  are  not  communicated  to  offerors 
in  the  RFP.  For  example,  under  the  "Technical"  factor,  the  subfactor  "Technical 
Approach"  is  communicated  in  the  RFP  as  being  the  most  important  relative  to  all  the  other 
"Technical"  subfactors.  "Technical  Approach"  in  this  situation  requires  that  the  offeror 
outline  how  he  would  accomplish  a  series  of  tests  as  delineated  in  the  PWS.  The  subfactor 
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does  not  inform  the  offeror  of  the  the  w  eights  attached  to  each  test  nor  does  it  inform  the 
offeror  of  all  the  qualiiati\  e  measures  the  Government  will  use  to  assess  the  effectiveness 
of  the  offeror's  approach.  For  instance,  will  the  Government  prov  ide  a  higher  score  to  an 
offeror  for  accuracy  above  a  certain  process  specification,  and  if  so  how  much  more?  This 
type  of  information  is  not  provided  in  the  RFP  and  therefore,  an  offeror  has  to  use 
judgment  as  to  where  to  place  emphasis.  Upon  not  being  selected  for  award,  the  contractor 
claims  he  did  not  understand  the  rules  and  thereafter  loses  faith  in  the  process. 

Should  evaluation  criteria  and  their  w’eights  be  communicated  to  offerors  in  the 
RfT*  of  "best  value"  procurements?  On  the  one  hand,  communicating  the  evaluation  criteria 
and  their  weights  in  the  RFP  may  eliminate  feelings  of  distrust  by  providing  offerors  with 
objective  measures  of  what  the  Government  considers  most  important  in  an  offeror's 
proposal.  However,  all  Government  interviewees  posed  this  question  believe  that 
providing  all  criteria  with  weights  may  also  stifle  contractor  innovation  and  limit  flexibility 
of  Government  selection  authorities  w  ho  may  then  be  forced  into  a  mathematical 
formulation  to  determine  the  most  advantageous  procurement  decision. 
b.  "Best  Value"  Versus  Low  Cost 

Contractors  do  not  alwavs  believe  DoD  is  awarding  contracts  based  on  the 
"best  value"  concept.  Instead,  many  contractors  believe  that  DoD  continues  to  place  too 
much  emphasis  on  the  low  bid.  This  concern  was  raised  by  numerous  Government  and 
contractor  personnel  interviewed  during  research.  In  essence,  these  interview'ees  believe 
the  Government  does  not  always  "stay  the  course"  in  making  "best  value"  contract  awards 
but  rather  continues  to  favor  low'  cost  over  quality  in  the  selection  of  contracts  for 
engineering  and  technical  services. 

An  example  of  this  issue  was  discussed  with  the  researcher  by  a  senior  Navy 
acquisition  official  who  recalled  a  meeting  w  ith  several  contractors  who  were  bidding  on 
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Navy  RFPs  for  engineering  and  technical  services  which  advertised  the  payment  of  a 
premium  for  a  superior  technical  proposal.  Surprisingh’,  even  with  the  offer  of  premiums, 
the  contractors  had  decided  upon  a  strategy  of  submitting  the  low  bid  in  order  to  win  the 
contracts.  This  strategy  was  based  on  the  knowledge  that  defense  cutbacks  are  limiting  the 
funds  available  for  support  ser\'ices  contracts.  The  reasoning  follow  s  that  with  reduced 
funding  available,  even  the  advertisement  of  a  premium  w  ill  not  stop  the  Government  from 
awarding  to  the  technically  acceptable  low  offeror.  Under  this  premise,  the  senior 
acquisition  manager  stated,  "Contractors  are  reading  the  tea  leaves  of  reduced  funding  for 
defense.  We  (Government)  are  operating  like  w'e  still  hax  e  plenty  of  money."  The  abo\’e 
example  is  a  representation  of  the  distrust  many  contractors  harbor  regarding  "best  value" 
evaluations  in  favor  of  low  bid. 

2.  An  Opposing  View 

Skeptical  Government  personnel  believe  that  many  contractors,  who  claim  that 
they  do  not  trust  in  "best  value"  source  selections,  are  using  "best  \'alue"  as  an  excuse  for 
poorly  written  proposals  which  did  not  win  the  contract  award.  This  line  of  reasoning 
shifts  the  blame  for  non-award  to  the  Government  and  in  particular,  on  the  "best  value" 
source  selection  process.  Several  Government  inteiA'iewees  expressed  the  opinion  that 
contractors  are  very'  pleased  with  the  "best  value"  concept  when  it  leads  to  a  contract  award 
for  their  company.  However,  when  it  does  not  lead  to  a  contract  award,  they  claim  that  the 
process  is  flawed. 

3 .  Summary 

Contractor  distrust  of  the  "best  value"  source  selection  process  can  lead  to  gaming 
and  protest.  Both  cost  the  Government  in  terms  of  time  and  resources.  While  considering 
the  complaint  of  contractor  distrust,  a  Government  acquisition  manager  summed  up  the 
issue  as  follows: 
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In  some  instances,  all  the  words  in  the  world  will  not  make  a  contractor  belie\’e  the 
Government  is  choosing  the  highest  quality  on  a  "best  \  alue"  basis.  Contractors  ha\  e 
to  see  it  in  our  actions.  This  is  a  long  term  t\'pe  thing  although  industry  is  cry  ing  to  sec 
results  now. 

F.  CHAPTER  SUMMARY 

Chapter  IV  prox  ided  a  definition  and  analysis  of  the  four  most  frequently  mentioned 
problems  identified  during  this  research  on  the  "best  value"  source  selection  process  in  the 
procurement  of  engineering  and  technical  serv  ices.  The  chapter  focused  on:  1)  identifying 
evaluation  criteria  which  discriminate  between  offerors,  2)  the  technical  evaluation  process, 
3)  the  contractor  gaming  of  cost  and  technical  proposals  and  4)  contractor  distrust  of  the 
"best  value"  source  selection  process. 

Chapter  V  will  offer  conclusions  and  recommendations  from  this  thesis  research. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

This  chapter  will  discuss  conclusions  and  recommendations  reached  as  a  result  of  this 
research.  In  addition,  the  research  questions  presented  in  Chapter  1  are  answered  and 
recommendations  for  further  research  are  presented. 

B.  CONCLUSIONS 

1 .  The  Expression  "Best  Value"  is  Defined  in  Terms  of  an  Objective 
and  a  Process. 

The  expression  "best  value"  is  defined  both  in  terms  of  an  objective  and  a  process. 
As  an  objective,  "best  value"  is  defined  as  the  most  advantageous  procurement 
decision.  This  decision  can  be  based  on  low  price,  low  overall  cost  or  other  factors.  Other 
factors  include  technical  approach,  management  capabilities  and  past  performance. 

As  a  process,  "best  value"  is  defined  in  terms  of  a  source  selection,  w  here  all 
factors  including  cost  are  considered  prior  to  making  a  decision  on  contract  award.  If 
properly  conducted,  the  "best  value"  process  should  lead  to  the  most  advantageous 
procurement  decision.  To  achieve  optimum  success,  "best  value"  as  a  process  relies  on 
proper  definition  of  the  requirement,  thorough  acquisition  planning  and  a  disciplined 
technical  evaluation.  This  conclusion  is  discussed  in  Chapter  II. 

2.  The  "Best  Value"  Source  Selection  Process  is  Composed  of 
Objective  and  Subjective  Elements. 

Rating  systems  are  objective  tools  employed  to  assist  the  SSA  in  making  an  award 
decision.  Frequently  though,  the  SSA  is  required  to  employ  subjective  judgments  in 


84 


deciding  contract  award.  The  need  for  subjective  judgment  in  "best  value"  decisions  can 
never  be  completely  eliminated.  This  was  discussed  under  definitions  offered  for  "best 
value"  in  Chapter  III. 

3 .  There  are  no  Impediments  Stated  in  Acquisition  Regulations  Against 
Utilizing  "Best  Value"  Source  Selection  in  the  Procurement  of 
Engineering  and  Technical  Services. 

A  review  of  policy  guidance  in  Chapter  II  revealed  no  regulations  prohibiting  the 
employment  of  a  "best  value"  procurement  strategy.  Current  acquisition  guidance  is 
emphasizing  "best  value"  as  a  method  in  which  quality  as  well  as  price  factors  are 
considered  in  a  contract  award  decision.  "Best  (greatest)  value"  is  referenced  in  the  FAR, 
in  a  recent  change  to  the  DFARS  and  was  emphasized  in  a  22  March  1991  memo  by  the  the 
Assistant  Secretary  of  the  Navy,  the  Honorable  Gerald  Cann.  In  addition,  there  are 
numerous  agency  and  local  instructions  referring  to  the  "best  value"  procurements.  "Best 
value"  procurements  are  not  recommended  for  every  item  or  service.  In  the  area  of 
engineering  and  technical  serv  ices,  "best  value"  procurements  are  reserved  for  requirements 
which  dictate  the  complex  integration  of  personnel  for  the  accomplishment  of  complex 
PWSs. 

4.  There  is  Little  Standardization  in  the  Rating  Systems  Utilized  in  the 
Navy  to  Evaluate  Contractor  Proposals. 

As  discussed  in  Chapters  II  and  III,  Navy  activities  employ  point  systems, 
composite  point  systems  with  adjectival  or  color  descriptors  and  strictly  adjectival  systems 
to  evaluate  contractor  proposals  under  "best  value"  procurements.  Although  each  system 
possesses  advantages  and  disadvantages,  each  system  is  still  able  to  obtain  the  most 
advantageous  procurement  decision.  The  common  denominator  in  all  rating  systems  with 
regard  to  "best  value"  lies  in  the  judgment  of  the  reasoned,  and  many  times  subjective 
decision  of  the  SSA. 
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5.  Procurement  Personnel  Know  When  to  use  "Best  Value" 
Procurements  for  Engineering  and  Technical  Services. 

In  accordance  w  ith  a  variety  of  guidance  in  the  FAR,  DFARS,  ASN  (RDA)  "best 

value"  policy  memorandum  and  local  policy  instructions.  Navy  procurement  personnel  of 

all  grades  consistently  demonstrated  that  they  understood  when  to  use  a  "best  value" 

procurement  strategy.  This  was  e\'idenced  in  responses  received  to  Question  #3  in  Chapter 

111. 

6.  Factors  and  Subfactors  are  Generally  Standard  in  "Best  Value" 
Procurements  of  Engineering  and  Technical  Services.  Evaluation 
Criteria  Vary  Depending  on  the  Requirement. 

As  dicussed  in  Question  #4  of  Chapter  III,  Technical,  Management  and  Cost 
factors  and  related  subfactors  were  generally  standard  in  "best  value"  procurements  for 
engineering  and  technical  ser\'ices  throughout  DoD.  Evaluation  criteria  describing  the 
factors  are  tailored  to  what  is  considered  important  in  the  requirement.  Developing 
evaluation  criteria  which  differentiate  between  offerors  and  lead  to  the  most  advantageous 
procurement  decision  is  a  major  issue  in  the  acquisition  planning  process. 

7 .  Past  Performance  Data  are  Utilized  by  Most  Activities,  However  a 
Standardized  Format  for  Collection  or  Type  of  Data  Does  not  Exist. 

The  overwhelming  majority  of  activities  utilize  past  performance  data  as  an 

evaluation  criterion  in  "best  value"  procurements  for  engineering  and  technical  services. 

However,  many  activities  use  qualitative  measures  such  as  improvements  made  to  past 

deficiencies,  while  other  activities  utilize  only  objective  measures  of  past  performance  such 

as  labor  costs,  the  skill  mix  utilized  and  uncompensated  overtime  hours  used  on  past 

contracts.  Objective  measures  involve  experience  in  years,  systems  or  both.  Qualitative 

methods  evaluate  the  effectiveness  of  the  experience. 

There  also  exist  differences  in  how  the  data  are  collected.  Many  activities  have 

formalized  systems  where  questionnaires  are  mailed  to  ail  Government  activities,  who  in 
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the  past,  ha\  e  utilized  the  the  contractor  for  similar  ser\  ices.  Other  activities  utilized  less 
formal  methods  of  telephone  checks  on  past  performance  and  the  word  of  local  technical 
personnel  who  ha\  e  used  the  contractor  in  the  past. 

There  is  no  Na\  y  standard  for  determining  what  past  performance  data  are  to  be 
collected,  how  they  are  to  be  collected  and  how  they  are  to  be  utilized.  Various  methods 
were  encountered  during  this  research.  At  present,  there  exists  no  centralized  contractor 
rating  system  in  DoD  for  professional  services,  although  NAVSEA  has  instituted  a 
Contractor  Performance  Index  pilot  program  that  appears  to  have  promise.  This  conclusion 
refers  back  to  data  collected  from  Question#5  in  Chapter  III. 

8 .  Draft  RFPs  and  Presolicitation  Conferences  are  Rarely  Utilized  for 

"Best  Value"  Procurements  of  Engineering  and  Technical  Services. 

Draft  RFPs  and  presolicitation  conferences  are  not  required  for  every  "best  value" 
procurement  of  engineering  and  technical  services.  However,  when  any  reasonable  doubt 
exists  as  to  the  clarity  of  the  PWS  or  the  RFP,  and  time  permits,  then  it  is  prudent  to 
request  comments  on  a  procurement  through  the  use  of  a  draft  RFP  or  a  presolicitation 
conference.  The  basis  for  this  conclusion  was  discussed  under  Problem  #1  in  Chapter  IV. 

9 .  There  is  no  Standard  Guidance  on  how  to  Accomplish  a  Cost 

Realism  Analysis  for  "Best  Value"  Procurements. 

As  discussed  in  Chapter  II  and  under  Question  #8  of  Chapter  III,  acquisition 
regulations  contain  several  definitions  for  cost  realism  and  define  when  it  should  be 
employed,  but  there  exists  no  standard  guidance  on  the  mechanics  of  performing  a  cost 
realism  analysis  for  professional  services.  Currently,  contracting  activities  perform  many 
of  the  same  steps  used  during  cost  analysis  when  cost  and  pricing  data  are  available. 
However,  cost  realism  analysis  is  completed  in  a  competitive  environment  and  certified  cost 
and  pricing  data  are  not  available. 
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10.  Although  FAR  15.610  now  Permits  Awarding  Without  Discussions 
for  "Best  Value"  Procurements,  this  Provision  is  Rarely  Utilized. 

This  researcher  onl\’  encountered  one  "best  value"  procurement  of  engineering  and 

technical  services  awarded  without  discussions.  The  award  was  made  to  the  technically 

superior,  low  priced  offeror  and  other  offerors  were  not  susceptible  or  capable  of  being 

made  susceptible  for  award.  However,  a  supervisor}’  contracting  officer  at  a  NRCC 

estimated  that  now,  up  to  25%  of  all  contract  awards  made  at  that  particular  NRCC  are 

awarded  on  a  "best  value"  basis  without  discussions.  The  past  policy  of  awarding  w'ithout 

discussions  only  to  the  low  priced  offeror  was  changed  by  Section  802  of  the  1991 

National  Defense  Authorization  Act  and  implemented  in  the  FAR  in  April  1991.  It  is 

expected  that  the  number  of  contract  awards  without  discussions  will  continue  to  increase 

in  the  future  as  contracting  personnel  become  familiar  with  the  new  regulation. 

1 1 .  Honest  and  Upfront  Dialogue  Between  Requiring  and  Procurement 
Organizations  is  a  Major  Element  to  Successful  "Best  Value" 
Procurements. 

This  conclusion  was  drawn  repeatedly  at  every  activity  visited  during  the  research. 
In  order  to  achieve  the  most  advantageous  procurement  decision  through  the  use  of  "best 
value"  procurements,  a  relationship  of  trust  and  respect  must  be  fostered  between  the 
requiring  and  procurement  organizations.  A  strong  relationship  improves  "best  value" 
procurements  at  every  stage  of  the  process  from  acquisition  planning  through  contract 
award. 


C.  RECOMMENDATIONS 

1 .  Contracting  Activities  Should  Strive  to  Ensure  That  Their  Customers 
Understand  the  Differences  in  Source  Selection  Methods  Between 
"Best  Value"  Source  Selection  and  Selecting  the  Low  Cost 
Technically  Acceptable  Offeror. 

Requiring  organizations  play  a  major  role  in  the  selection  of  an  acquisition 
strategy.  By  understanding  the  differences  in  source  selection  methods,  requiring 
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organizations  who  understand  the  technical  requirements,  are  better  equipped  to  make  a 
recommendation  on  the  acquisition  strategy.  Depending  upon  the  complexity  of  the 
services  required  and  the  evaluation  criteria,  either  procurement  method  may  lead  to  the 
most  advantageous  procurement  decision. 

2 .  The  Acquisition  Planning  Team  and  Technical  Evaluation  Committee 
Should  Institute  Feedback  Mechanisms  for  the  Improvement  of 
Evaluation  Criteria  Used  in  "Best  Value"  Procurements. 

This  recommendation  would  assist  both  technical  and  procurement  organizations  in 

the  planning  and  evaluation  of  future  acquisitions.  Ideally,  after  award  of  a  "best  value" 

procurement,  the  acquisition  planning  team  and  the  technical  evaluators  should  meet  to 

debrief  the  procurement.  Specific  areas  to  be  addressed  would  focus  on  the  evaluation 

factors,  criteria,  TEC  composition  and  any  area  where  uncertainties  arose  during  the 

evaluation.  A  problem  uith  this  recommendation  is  the  investment  of  time  required  to 

conduct  such  a  debriefing.  However,  if  properly  utilized,  a  debriefing  of  the  procurement 

may  save  time  and  lead  to  more  effective  "best  value"  procurements  in  the  future. 

3 .  Contracting  Officers  and  Negotiators  Should  be  Provided  Abstracts 
of  Comptroller  General  and  BCA  Decisions  Germane  to  "Best  Value" 
Procurements. 

NAVSUP  should  synopsize,  or  designate  a  field  contracting  activity  to  synopsize. 
Comptroller  General  and  BCA  decisions  germane  to  "best  value"  procurements.  These 
synopses  should  then  be  distributed  to  field  contracting  organizations  for  their  information 
and  education.  This  knowledge  will  help  improve  the  "best  value"  source  selection 
process.  Synopsizing  the  decisions  will  ensure  that  the  key  issues  in  the  cases  are 
highlighted  and  save  on  the  time  required  to  read  and  comprehend  a  decision.  Synopses 
should  include  only  relevant  facts  of  the  cases  without  further  interpretation. 
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4.  Emphasize  Management  Dialogue  Between  Contracting  and  Technical 
Organizations. 

This  recommendation  is  basic  to  all  contrac'mg  and  technical  organizations  and  is 
presently  followed  in  varying  degrees.  However,  based  on  this  researcher's  findings, 
more  effort  is  required.  The  technical  and  contracting  organizations  must  emphasize  the 
use  of  honest  dialogue  at  all  stages  of  a  "best  value"  procurement.  The  key  to  success  is  in 
ensuring  that  each  organization  understands  the  requirements  of  the  other.  This  is 
especially  true  in  the  areas  of  acquisition  planning  and  technical  evaluations  of  proposals. 
In  the  area  of  technical  evaluations,  it  was  mentioned  in  this  thesis  that  the  narrative  in  the 
technical  evaluations  are  consistently  inadequate.  Continued  dialog  between  the  two 
activities  could  help  correct  this  problem. 

5 .  The  Customer  Activity  Must  Dedicate  Highly  Experienced  Personnel 
to  the  Technical  Evaluation  Committee  or  Risk  Living  with  a 
Mediocre  Service  Contractor  for  Years  in  the  Future. 

One  of  the  major  deficiencies  noted  in  the  technical  evaluation  process  was  a  lack 

of  qualified  personnel  to  conduct  the  technical  evaluation.  The  management  at  the  requiring 

activity  should  ensure  that  experienced  personnel  are  assigned  to  the  technical  evaluation 

committees.  In  addition,  technical  evaluators  should  be  relieved  of  all  other  duties  and  be 

evaluated  on  their  performance  as  a  member  of  the  technical  evaluation  committee.  The 

management  at  the  requiring  activities  should  also  seek  innovative  methods  of  assigning 

persons  to  the  technical  evaluation  committee  such  as  a  jury  duty  type  arrangement  for  a 

fixed  period  of  time. 

Contracting  organizations  should  recognize  and  be  sensitive  to  the  time  and 
personnel  constraints  at  the  technical  organizations.  The  contracting  organization  can  assist 
by  helping  to  streamline  the  acquisition  process  and  be  available  for  questions  during  the 
technical  evaluation.  Contracting  organizations  and  technical  organizations  should  also 
consider  utilizing  technical  managers  as  the  SSAs  during  procurements. 
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6 .  The  Navy  Should  Consider  Implementing  a  Standardized  Rating 
System  for  use  in  Evaluating  Contractor  Proposals. 

There  exists  a  variety  of  rating  systems  used  for  "best  value"  procurements  at 

contracting  organizations  throughout  the  Navy.  Point  systems,  composite  point  systems 

with  adjectival  or  color  descriptors  and  strictly  adjectival  systems  with  no  points  are  all 

currently  used  to  rate  contractor  proposals  for  "best  value"  procurements.  Standardizing  a 

rating  system  throughout  the  Na\  y,  may  improve  the  efficiency  and  effectiveness  of  "best 

value"  procurements  in  the  long  run  because  all  concerned.  Government  and  industry,  will 

be  familiar  with  the  standard.  On  the  other  hand,  a  standard  rating  system  may  stifle 

innovative  activities  from  experimenting  with  new  systems  which  could  be  tailored  for  their 

needs. 

7 .  The  Navy  Should  Provide  Standardized  Guidance  on  how  to  Perform 
a  Cost  Realism  Analysis  for  Engineering  and  Technical  Services. 

'  A  regulation  or  instruction  on  how  to  perform  cost  realism  analysis  is  not 

recommended.  What  is  recommended  is  basic  guidance  similar  to  that  available  at  NRCC 

San  Diego,  Lxmg  Beach  Detachment  which  first  defines  cost  realism  and  presents  scenarios 

in  line  with  past  cost  realism  situations.  The  goal  of  this  guidance  should  not  be  focused 

on  "how  to"  but  rather  to  sensitize  the  cost  and  technical  evaluators  to  cost  realism  issues. 

8 .  Key  Personnel  Salaries  and  Other  Cost  Information  Should  be 
Integrated  into  Technical  Evaluations. 

Salary  and  other  cost  information  relevant  to  the  technical  proposal  should  be 
integrated  into  the  technical  evaluation  to  help  ensure  that  the  costs  are  realistic,  protecting 
both  the  Government  and  responsible  contractors  from  unrealistic  rates.  This  information 
should  be  compared  to  the  level  of  experience  of  the  individuals  and  the  technical  approach 
proposed.  Problems  with  maintaining  the  privacy  of  contractor  {personnel  proposed  could 
be  averted  by  providing  a  salary  range  versus  a  fixed  figure. 
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9.  Contracting  Organizations  Should  Streamline  the  "Best  Value" 
Procurement  Process  by  Emphasizing  the  use  of  Award  Without 
Discussions. 

Making  an  award  u  ithout  discussions  permits  the  contracting  acti\'ity  to  streamline 
the  source  selection  process  by  eliminating  the  requirement  for  discussions  and  subsequent 
BAFOs.  The  key  to  awarding  without  discussions  begins  with  good  acquisition  planning 
and  continues  through  a  disciplined  evaluation  process.  Making  an  award  without  the 
requirement  for  discussions  should  be  one  of  the  foremost  objectives  of  the  contracting 
organization  since  it  can  significantly  streamline  the  process.  Thoughtful  scenarios  should 
be  developed  and  discussed  at  each  activity  as  to  when  awarding  without  discussions  on  a 
"best  value"  basis  w'ould  be  applicable. 

10.  Contracting  Activities  Should  Institute  Formalized  Methods  of 

Collecting  and  Evaluating  Qualitative  Past  Performance  Data  for  use 
in  "Best  Value"  Procurements. 

Contractors  need  to  understand  that  the  quality  of  past  contract  performance  will  be 
one  of  the  evaluation  criteria  for  all  "best  value"  procurements.  One  method  of 
disseminating  this  policy  to  contractors,  is  to  implement  formalized  procedures  to  collect 
and  evaluate  past  performance  data  As  mentioned  in  the  conclusions,  there  exists  a  variety 
of  collection  methods  and  tyf)es  of  data  utilized  in  the  area  of  f>ast  performance  evaluations. 
A  standardized  Navy  policy  is  not  recommended.  However,  formalized  approaches 
stressing  qualitative  assessments  of  past  performance  appear  to  have  the  most  promise  in 
ensuring  contractors  understand  the  importance  of  quality  contract  performance. 

D.  ANSWERS  TO  RESEARCH  QUESTIONS 

1 .  Primary  Research  Question:  What  is  the  Most  Efficient  and  Effective 
Method  of  Implementing  a  "Best  Value”  Procurement  Strategy  for 
Engineering  and  Technical  Services  Contracts? 

There  is  no  one  specific  method  for  implementing  "best  value"  that  is  most  efficient 
or  effective.  However,  there  are  basic  tenets  to  consider  in  every  "best  value" 
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procurement.  "Best  value"  procurements  are  accomplished  within  a  framework  of:  1) 
properly  identifying  the  requirement,  2)  preparing  clear  and  relevant  evaluation  criteria,  3) 
conducting  a  disciplined  evaluation  process  and,  4)  determining  the  most  advantageous 
procurement  decision  to  the  Government.  This  framew  ork  requires  continuous  planning 
and  evaluation  at  e^•er^•  stage  to  impro\  e  the  efficiency  and  effectiveness  of  the  process. 

This  research  highlighted  specific  areas  for  contracting  organizations  to  review  prior  to 
and  during  "best  value"  procurements  w  hich  include  the: 

1.  Training  and  communications  between  the  members  of  the  acquisition  planning 
organization. 

2.  Need  for  the  use  of  draft  RFPs  and  presolicitation  conferences  to  solicit  industry 
input. 

3 .  Rating  systems  used  by  the  organization. 

4.  Training  and  motivation  of  the  technical  evaluation  committee. 

5.  The  cost  realism  analysis. 

6.  Debriefing  or  lessons  learned  process  after  contract  award. 

2.  Subsidiary'  Question  1:  What  is  "Best  Value"? 

For  purposes  of  this  research,  "best  value"  was  defined  in  terms  of  an  objective 
and  a  process.  As  an  objective,  "best  value"  represents  the  most  advantageous 
procurement  decision  available  to  the  Government.  This  decision  is  based  on  low  price, 
low'  overall  cost  or  other  factors.  "Best  value"  defined  as  an  objective  has  extremely  broad 
applications.  One  can  obtain  the  "best  N  alue"  under  any  source  selection  method  as  long  as 
the  product  or  sen  ice  meets  the  criteria  of  providing  optimum  satisfaction  of  mission 
needs. 

"Best  value"  is  also  defined  in  terms  of  a  source  selection  evaluation  process  where 
all  relevant  factors  are  taken  under  consideration  prior  to  making  an  aw'ard  decision.  The 
end  result  of  this  process  represents  the  most  advantageous  procurement  decision  available. 
In  this  context,  procurement  decisions  are  not  based  solely  on  price. 

As  a  process,  "best  value"  directly  contrasts  with  technically  acceptable  low  offeror 
source  selections  which  make  contract  awards  based  on  the  low  price  among  technically 


acceptable  offerors.  "Best  value"  emphasizes  other  factors  ir\  addition  to  price  and 
establishes  different  levels  of  technical  merit  from  which  to  make  the  most  advantageous 
procurement  decision. 

3.  Subsidiary  Question  2:  Under  What  Circumstances  is  "Best  Value" 
Employed? 

"Best  value"  is  generally  reserved  for  the  procurement  of  complex  goods  and 
services  where  alternate  approaches  for  satisfying  a  requirement  are  available.  This  is  due 
to  the  often  times  subjecti\’e  nature  of  a  "best  value"  procurement  where  evaluation  factors 
and  criteria  are  utilized  to  discriminate  betw'een  solutions  offered  by  competing  proposals. 
As  a  result,  requirements  necessitating  the  use  of  performance  specifications  or  complex 
services  are  typically  considered  for  "best  value"  procurements.  Conversely,  products  with 
detailed  specifications  and  routine  seix'ices  are  acquired  based  on  low  cost. 

4.  Subsidiary  Question  3:  What  Source  Selection  Methods  are  Used  for 
the  "Best  Value"  Procurement  of  Engineering  and  Technical 
Services? 

Two  source  selection  methods  are  typically  used  for  "best  value"  procurements  of 
engineering  and  technical  services.  Those  methods  are  formal  source  selection  and 
streamlined  or  informal  source  selection.  Both  methods  utilize  evaluation  factors  to 
differentiate  between  offerors.  Formal  source  selection  is  administratively  expensive  and  is 
generally  reserved  for  complex  or  higher  cost  requirements.  The  streamlined  source 
selection  method  is  more  common  in  practice.  Streamlining  is  normally  accomplished  by: 
1)  limiting  the  number  of  pages  in  a  proposal,  2)  reducing  the  number  of  evaluation  factors 
and/or  3)  reducing  the  number  of  personnel  involved  in  proposal  evaluation. 
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5 .  Subsidiary  Question  4:  What  Factors  and  Subfactors  are  Utilized  in 
"Best  Value"  Procurements  for  Engineering  and  Technical  Services? 

From  a  sample  taken  during  this  research,  evaluation  factors  and  subfactors  used 

in  "best  value"  evaluations  are  fairly  standard  in  title  and  content.  The  factors  used  include 

technical,  management  and  cost.  Technical  and  cost  were  factors  in  all  "best  value" 

procurements  for  engineering  and  technical  sen  ices.  Management  was  primarily  listed  as  a 

subfactor  of  technical.  Subfactors  under  the  above  factors  include  personnel  experience, 

technical  approach,  management  plan,  manpower  utilization  matrix,  sample  tasks  and  past 

performance. 

6 .  Subsidiary  Question  5:  What  are  the  Problems  or  Impediments  to 
Implementing  "Best  Value"  Source  Selection  for  Engineering  and 
Technical  Services? 

Several  problems  and  impediments  were  identified  during  the  course  of  this 
research.  The  majority  of  the  problems  and  impediments  were  in  the  areas  of  acquisition 
planning  and  proposal  evaluation.  Specific  problems  mentioned  most  often  during 
interviews  included;  1)  the  identification  of  evaluation  criteria  which  will  differentiate 
between  offerors  and  lead  to  the  most  adx  antageous  procurement  decision,  2)  the  technical 
evaluation  process,  3)  contractor  gaming  of  cost  and  technical  proposals  and,  4)  contractor 
distrust  of  the  "best  value"  procurements  which  advertise  technical  ability  over  cost. 
Impediments  to  correcting  the  problems  pervasively  centered  on  a  lack  of  time,  training, 
communications  and  resources  necessary  to  plan,  document  and  conduc,  a  "best  value" 
procurement. 

7 .  Subsidiary  Question  6:  How  do  Activities  Quantify  Paying  a 
Premium  on  "Best  Value"  Procurements? 

All  activities  questioned  emphasized  reasoned  judgment  prior  to  extending  a 
premium  to  a  contractor  for  a  superior  technical  proposal.  However,  most  activities  also 
conduct  a  cost  benefit  analysis  in  which  additional  technical  points  are  equated  to  additional 
dollars.  The  cost  benefit  analysis  is  carried  out  by  the  technical  evaluators,  who  at  certain 
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activities,  are  provided  the  cost  differentials.  Other  activities  do  not  supply  the  technical 
evaluators  with  a  cost  differential.  In  these  cases  technical  evaluators  are  provided  with  the 
differences  in  technical  proposals  and  are  asked  to  justify’  the  diflerences  without  the  dollar 
differential  at  hand. 

E.  RECOMMENDATIONS  FOR  FURTHER  RESEARCH 

1 .  Conduct  a  cost  benefit  analysis  of  uncompensated  o\  ertime  use  for  engineering 
and  technical  ser\’ices  at  one  acti\  ity. 

2.  Investigate  the  most  efficient  and  effective  method  to  collect  and  evaluate  past 
performance  data  for  use  in  the  source  selection  of  professional  ser\’ices  contracts. 

3.  Determine  the  elements  for  and  construct  a  "guide"  on  the  conduct  of  a  cost  realism 
analysis  for  professional  serv  ices  procurement  at  NAVSUP  field  activities. 

4.  Evaluate  the  technical  proposal  evaluation  process  for  engineering  and  technical 
services  at  two  field  acti\  ities.  Compare  and  contrast  the  results. 

5.  Study  the  Acquisition  Review  Program  (ARP)  at  NWC  China  Lake,  California  and 
make  recommendations  for  improvement  and  export  to  other  activities. 

6.  Define  the  applicability  of  making  contract  awards  w'ithout  discussions  in  "best 
value"  procurements.  Present  a  background  and  scenarios  where  awarding  w'ithout 
discussions  may  streamline  the  "best  value"  procurement  process. 

7.  Investigate  rating  systems  used  in  "best  value"  procurements  and  decide  whether 
the  Navy  should  implement  a  standardized  rating  system. 
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APPENDIX  A 


THE  ASSISTANT  SECRETARY  OF  THE  NAVY 
(Research,  Development  and  Acquisition) 

WASHINGTON,  D.C.  20350-1000 

MAR  22  1991 

MEMORANDUM  FOR  PROGRAM  EXECUTIVE  OFFICERS 

DIRECT  REPORTING  PROGRAM  MANAGERS 
COMMANDERS  OF  SYSTEMS  COMMANDS 
DEPUTY  CHIEF  OF  STAFF  FOR  INSTALLATIONS  ANT) 
LOGISTICS,  HEADQUARTERS,  U.S.  MARINE  CORPS 
COMMANDING  GENERAL,  MARINE  CORPS 
RESEARCH,  DEVELOPMENT  AND  ACQUISITION 
COMMAND 

COMMANDER,  MILITARY  SEALIFT  COMMAND 
COMMANDING  OFRCER,  AUTOMATIC  DATA 
PROCESSING  SELECTION  OFFICE 
COMMANDING  OFFICER,  NAVAL  COMMERCIAL 
COMMUNICATIONS  CENTER 

Subj:  "BEST  VALUE"  CONTRACTING  POLICY  (DCN  I82I0021/APIA-0I-000I) 

The  purpose  of  this  memorandum  is  to  reemphasize  the  Navy's  commitment  to 
competitively  select  sources  on  the  basis  of  the  "best  value"  to  the  Government.  In 
determining  factors  to  be  e\'aluated  other  than  price,  the  nature  of  the  supplies  or  ser\'ices  to 
be  acquired  must  be  considered. 

Factors  impacting  "best  value"  decisions  include  technical  compjetence,  proven  past 
performance,  management  capability,  life  cycle  costs  and  quality.  Evaluation  of  these 
factors  should  be  structured  to  ensure  that  they  are  given  due  consideration  in  determining 
the  overall  benefit  associated  with  the  offered  price. 

"Best  value"  evaluations  should  be  employed  wherever  possible  and  especially  in 
weapons  system,  federal  information  processing  (FIP)  and  professional  and  technical 
service  support  acquisitions.  Requirements  which  dictate  complex  integration  of  people, 
equiinnent,  hardware,  innovation  and  software  should  also  be  evaluated  on  a  "best  value" 
basis.  However,  when  non-complex,  routine  requirements  are  being  procured  and  the 
|»oduct  to  be  delivered  is  clearly  defined  at  the  outset  of  the  procurement,  it  may  be 
api»‘c^riate  to  award  to  the  lowest  priced,  technically  acceptable  offeror. 

nease  circulate  this  memorandum  to  the  apprqjriate  personnel  as  it  reemphasizes  the 
Navy's  commitment  to  "best  value"  when  acquiring  systems,  products  and  services  for  the 
Fleet. 


Gerald  A.  Cann 
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APPENDIX  B 


LIST  OF  INTERVIEWEES 


The  follow  ing  is  a  list  of  persons  w  ho  directly  provided  information  for  this  research. 

Interviews  consisted  of  both  telephone  conversations  and  personal  x  isits. 

1)  Ayars,  J.,  Deputy  Director  of  Contracts,  Naval  Supply  Center,  Puget  Sound,  WA, 
July  30,  1991. 

2)  Barker,  W.,  Director  Information  Resources  Contracts  Division,  Naval  Regional 
Contracting  Center,  Washington,  DC,  September  11,  1991. 

3)  Bugby,  D.,  Director  of  Business  Government  Policy,  Professional  Services 
Council,  Washington,  DC,  August  1,  1991. 

4)  Camph,  B.,  Manager  of  Contracts,  SRI  International,  Menlo  Park,  CA,  various, 
September  1991. 

5)  Canales,  M.,  Contracting  Officer,  PMA  Services,  Naval  Air  Systems  Command, 
Washington,  DC,  September  11,  1991. 

6)  Canfield,  T.,  Capt.,  SC,  USN,  Director  of  Command  Inspection  Division,  Navy 
Inspector  General  Staff,  Washington,  DC,  July  28,  1991. 

7)  Carberr>,  V.  S.,  Executive  Director  of  Contracts  Management,  Na\’al  Air  Systems 
Command,  Washington,  DC,  September  11, 1991. 

8)  Celloto,  D.,  Deputy  Counsel,  Naval  Regional  Contracting  Center,  Philadelphia, 
PA,  August  2,  1991. 

9)  Cengia,  R.,  Supervisor)’  Contracting  Officer,  Naval  Regional  Contracting  Center, 
San  Diego  Detachment,  Long  Beach,  CA,  various,  July  -  November,  1991. 

10)  Cook,  O.,  Contracting  Officer,  CTA  Systems  Support  Contract,  China  Lake,  CA, 
August  13,  1991. 

11)  Finnigan,  M.,  Manager.  Senices  Contract  Division,  Naval  Aviation  Center, 
Indianapolis,  IN,  August  7,  1991. 

12)  Ford,  F.,  Office  of  Assistant  Secretary  of  the  Navy,  Washington,  DC,  various, 
August  -  October,  1991, 

13)  Hallum,  L.,  Contract  Cost/Price  Analyst,  Directorate  of  Contracting,  Fort  Ord, 
Monterey,  CA,  various,  July  -  August,  1991. 
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14)  Hart,  N.,  Cdr.,  SC,  USN,  Director  of  Contracts,  Naval  Supply  Center,  Puget 
Sound,  WA,  July  27,  1991. 

15)  Hauf,  B.,  Capt.,  USAF,  Air  Force  Logistics  Management  Center,  Gunter  Air 
Force  Base,  AL,  September  25,  1991. 

16)  Hauenstein,  W.,  Radm.,  SC,  USN,  Competition  Advocate  of  the  Na\  3  ,  Office  of 
the  Assistance  Secretary  of  the  Navy,  Washington,  DC.,  August  1,  1991. 

17)  John,  W.,  Division  Director,  Missile  and  Systems  Contracts  Division,  Naxal  Air 
Systems  Command,  Washington,  DC,  September  11,  1991. 

18)  Kaczmarek,  T.,  Senior  Contract  Negotiator,  Naval  Regional  Contracting  Center, 
Philadelphia,  PA,  July  18,  1991. 

19)  Kalapxrs,  M.,  Capt.,  SC,  USN,  Director  of  Contracts,  Na\  al  Training  Systems 
Center,  Orlando,  FL,  July  27,  1991. 

20)  Kaufman,  S.,  Deputy  Associate  Administrator,  Office  of  Federal  Procurement 
Policy,  Washington,  DC,  August  8,  1991. 

21)  Kolis,  T.,  Procurement  Analyst,  National  Aeronautics  and  Space  Administration, 
Ames,  CA,  July  10,  1991. 

22)  LaFollette,  C.,  Chief,  Management  Branch  for  Engineering  and  Technical 
Services,  National  Aeronautics  and  Space  Administration,  Ames,  CA,  various, 
August  -  September,  1991. 

23)  Layering,  F.,  Program  Manager,  ADP  and  Telecommunications  Support  Systems, 
Boeing  Computer  Serv  ices,  China  Lake,  CA,  August  13,  1991. 

24)  Manson,  B.,  Deputy  Director  of  Contracts,  Naval  Weapons  Center,  China  Lake, 
CA,  various,  August  -  November,  1991. 

25)  McCabe,  T.,  Chief,  Contract  Div  ision,  Defense  Nuclear  Agency,  Alexandria,  VA, 
August  28,  1991. 

26)  McCarthy,  P.,  Lcdr,  SC,  USN,  Contracts  Section  Head,  Naval  Sea  Systems 
Command,  Washington,  DC,  various,  July  -  October  1991. 

27)  Monckton,  W.,  Contract  Specialist,  Naval  Weapons  Center,  China  Lake,  CA, 
August  14,  1991. 

28)  Moris,  M.,  Chief  of  Contracting  Section,  Department  of  the  Army  Central 
Contracting  Office,  Atlanta,  GA,  July  12,  1991. 

29)  Murphy,  B.,  Head,  Radar  and  Fire  Control  Branch,  Contracts  Directorate,  Naval 
Sea  Systems  Command,  Washington,  DC,  July  17,  1991. 

30)  Norman,  R.  Vice  President  of  Contracts  and  Administration,  ROH  Inc., 
Alexandria,  VA,  July  20,  1991. 
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31)  Passar,  A.,  Maj.  USA,  Contract  Law  Attorney,  Department  of  the  Army  Central 
Contracting  Office,  Atlanta,  GA,  Jul j  30, 1991 . 

32)  Patterson,  J.,  Contract  Specialist,  Naval  Weapons  Center,  China  Lake,  CA, 
August,  12,  1991. 

33)  Paul,  D.,  Contract  Specialist,  Naval  Weapons  Center,  China  Lake,  CA,  August  15 
1991. 

34)  Penn,  A.,  COTR,  Boeing  ADP  Support  Services  Contract,  Naval  Weapons 
Center,  China  Lake,  CA,  August  13,  1991. 

35)  Reese,  B.,  Director  of  Contracts,  Techmatics  Corporation,  Alexandria,  VA,  Julv 
27,  1991. 

36)  Richter,  R.,  Assistant  for  Engineering  Programs,  Naval  Ship  Weapon  Systems 
Engineering  Station,  Port  Hueneme,  CA,  July  17,  1991. 

37)  Rainwater,  S.,  Ordering  Officer,  Boeing  ADP  Support  Services  Contract,  Naval 
Weapons  Center,  China  Lake,  CA,  August  14-15,  1991. 

38)  Riggs,  R.,  Associate  Department  Head,  Engineering  Department,  Naval  Weapons 
Center,  China  Lake,  CA,  August  14, 1991. 

39)  Rodgers,  D.  Supervisor^'  Contracting  Officer,  Naval  Regional  Contracting  Center, 
San  Diego  Detachment,  Long  Beach,  CA,  June  25, 1991. 

40)  Rumberger,  R.,  Branch  Head,  Systems  Control  Branch,  Naval  Air  Systems 
Command,  Washington,  DC,  September  12,  1991. 

41)  Ryan,  J.  Cdr.,  SC,  USN,  Deputy  Division  Director,  Naval  Air  Systems 
Command,  Washington,  DC.,  various,  August  -  November,  1991. 

42)  Siminkski,  M.,  Chief  of  Contract  Operations  Directorate,  Vandenburg  Air  Force 
Base,  CA,  August  29,  1991. 

43)  Sueur,  R.,  Cdr.,  SC,  USN,  Commanding  Officer,  Naval  Regional  Contracting 
Center,  San  Diego  Detachment,  Long  Beach,  CA,  June  24,  1991. 

44)  Safford,  D.,  Legal  Counsel,  Naval  Regional  Contracting  Center,  San  Diego 
E)etachment,  Long  Beach,  CA,  June  25,  1991. 

45)  Swizewski,  J.,  Supervisory  Contract  Negotiator,  Naval  Regional  Contracting 
Center,  Philadelphia,  PA,  August  2, 1991. 

46)  Tarves,  A.,  Supervisory  Contract  Negotiator,  Naval  Regional  Contracting  Center, 
Philadelphia,  PA,  July  30,  1991. 

47)  Taylor,  S.,  Supervisory  Contracting  Officer,  Naval  Regional  Contracting  Center, 
Washington,  DC,  September  11, 1991. 
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48)  Trunk,  M.,  Procurement  Policj-  Analyst,  Naval  Supply  Systems  Command, 
Washington,  DC,  various,  July  -  October  1991. 

49)  Utendorff,  D.,  Technical  Assistant  to  the  Deputy  Chief  of  Staff  for  Contracting, 
Aeronautical  Systems  Division,  Wright  Patterson  Air  Force  Base,  Dayton,  OH, 
August  28,  1991. 

50)  Vollmer,  J.,  Project  Director,  EGG  Inc.,  Alexandria,  VA,  July  17,  1991. 

51)  Webb,  D.,  Contracting  Officer,  Naval  Regional  Contracting  Center,  San  Diego 
Detachment,  Long  Beach,  CA,  June  25,  1991. 

52)  Wice,  M.,  Ordering  Officer,  COMARCO  Support  Services  Contract,  Naval 
Weapons  Center,  China  Lake,  CA,  August  11,  1991. 

53)  Williams,  C.,  Deputy  Director,  Onsite  Field  Support  Office,  Naval  Supply 
Systems  Command,  Washington,  DC,  July  22,  1991. 
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APPENDIX  C 


SURVEY  QUESTIONNAIRE 


Government: 

1 .  How  would  you  define  "best  (greatest)  value?" 

2.  Does  your  activity  use  numerical  scoring,  adjective  grades  or  colors  to  e\  ''>uate 
proposals  to  achie\'e  "best  value"?  Why  is  that  rating  system  used? 

3.  Hov\'  do  you  decide  a  whether  certain  requirement  requires  the  usage  of  the  "best 
value"  method? 

4.  What  are  the  factors  and  subfactors  your  activity  uses  for  "best  value"  procurements 
of  engineering  and  technical  services  contracts? 

5.  How  is  past  performance  data  collected  and  utilized  in  "best  \’alue"  source  selection 
for  engineering  and  technical  services  contracts? 

6.  What  are  some  of  the  problems  or  issues  your  activity  faces  with  "best  value" 
procurements  of  engineering  and  technical  services  contracts? 

7.  How'  does  your  activity  determine  the  need  to  pay  a  premium  for  a  technically 
superior  proposal  o\’er  a  proposal  which  is  technically  acceptable,  yet  lower  priced? 

8.  How  does  your  activity  conduct  cost  realism  analysis  for  engineering  and  technical 
ser\  ices  contracts? 

Contractor. 

1 .  How'  would  you  define  "best  (greatest)  value?" 

2.  How  do  you  feel  about  the  "best  value"  concept  utilized  by  the  Government  in 
awarding  contracts? 

3.  Does  your  firm  have  any  problems  with  "best  value"  solicitations  let  by  DoD  for 
engineering  and  technical  services?  If  so,  what  are  they? 

4.  How  do  you  feel  about  awarding  contracts  on  a  "best  value"  basis  without 
discussions? 

5.  In  your  view,  does  the  Government  do  an  adequate  job  of  cost  realism  analysis? 
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APPENDIX  D 


CONTRACTOR  EVALUATION  QUESTIONNAIRE 

Date:  _ 

Contract#:  _  CONTRACTOR:  _ 


Option  period  being  evaluated: 


COTR  Name: 


Code: 


Phone: 


The  Rating  System  is  as  Follows: 

O  -  Outstanding 
B  -  Better 
A  -  Acceptable 


M  -  Marginal 
U  -  Unacceptable 


Technical  Approach 

Did  the  contractor  perform  as  required?  O 

How  well  did  the  contractor  continuall>'  improve  his 
approach  in  solving  technical  problems?  O 

Schedule 

How  well  did  the  contractor  make  delivery  of  CDRL 
items  in  accordance  with  the  terms  and  conditions? 

Was  rework  required?  O 

How  well  did  the  contractor  demonstrate  organizational 
methods  to  recognize  possible  delays  and  take  corrective 
actions  immediately?  O 

How'  well  did  the  contractor  keep  the  Government 
adequately  informed  of  schedule  issues.  O 

Management  Approach 

How  well  did  the  contractor  maintain  an  open  line  of 
communication  so  that  the  requiring  code  was  apprised 
of  technical,  cost  and  schedule  issues?  O 


B  A  M  U 

B  A  M  U 

B  A  M  U 

B  A  M  U 

B  A  M  U 

B  A  M  U 
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How  well  did  the  contractor  show  flexibility  and 
ingenuity  in  problem  soh  ing? 


O 


A 


U 


B 


M 


How  well  did  the  contractor  pro\  ide  clear,  concise 
and  informative  written  reports? 

Cost 

How  well  did  the  contractor  prov’ide/recommend  a 
reasonable  labor  mix? 


O  B  A  M  U 

O  B  A  M  U 


Personnel 

Did  the  contractor  have  a  stable  work  force?  O  B  A  M  U 


Did  the  contractor  prov  ide  a  quality  work  force? 


O  B  A  M  U 


Endorsement 

Based  on  the  experience  you  have  had  w'ith  this  contractor,  w'ould  you  elect  to  have  them 
perform  a  contract  for  you  again? 

Please  indicate  along  the  scale  your  overall  rating  of  the 

contractor's  performance  under  this  contract?  O  B  A  M  U 

General  Information 

What  was  the  contractor's  greatest  strength? 


What  w  as  the  contractor's  greatest  w  eakness? 


PR# _  will  be  placed  under  the  new 

option  year 

Estimated  Hours _ 

Estimated  Price _ 
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APPENDIX  E 


NSC  PUGET  SOUND  TECHNICAL  EVALUATOR  TEST 

1 .  Under  what  circumstances  should  Source  Selection  Procedures  be  used? 

2.  Who  develops  the  evaluation  plan,  and  once  developed,  what  is  the  next  step? 

3.  The  plan  must  include  six  major  factors.  What  are  they,  and  discuss/describe  each 
one. 

4.  What  are  the  duties  of  the  Evaluation  Committee  Chairperson? 

5.  Who  should  be  on  the  E\  aluation  Committee,  and  what  are  their  functions? 

6.  How  are  the  Source  Selection  plan  criteria  used  by  the  PCO  in  the  solicitation? 

7.  Evaluation  criteria  must  be  in  concert  with  the  statement  of  work.  The  criteria  must 
also  possess  certain  other  characteristics.  Indicate  whether  the  follou  ing  statements  are 
true  or  false. 

a.  Criteria  should  overlap  each  other  as  littles  as  possible. 

b .  You  should  develop  as  many  criteria  as  possible  so  that  scoring  and  ranking  can 
be  broken  down  as  much  as  possible. 

c.  You  should  include  criteria  that  all  offerors  are  likely  to  meet  so  that  only 
formulas  or  definitive  rules  can  be  used  to  evaluate,  and  to  ensure  that  all  offerors  will 
receive  some  points. 

d.  Criteria  should  not  be  so  narrowly  defined  that  evaluation  of  responses  becomes 
a  yes/no  decision. 

e.  Criteria  should  be  worded  as  generally  as  possible  to  allow  various 
interpretations  of  the  evaluation  plan  in  order  that  varied  responses  will  be  received  from 
offerors. 

f.  Criteria  should  challenge  the  offeror  to  describe  a  technical  approach  in  sufficient 
detail  to  that  meaningful  differences  among  offerors  will  be  brought  out  through  the 
evaluation  process. 

8.  The  plan  may  include  mandatory  or  "non-negotiable"  standards.  What  are  some  of 
the  dangers  of  using  these,  and  how  must  they  be  used? 

9.  In  the  evaluation  process,  may  offerors  be  compared  to  each  other?  Discuss  your 
answer. 
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10.  The  most  commonly  used  approach  to  ranking/rating  proposals  is  the 
narrative/numerical  methodology.  Describe  the  process. 

11.  In  this  example,  the  Source  Selection  Plan  contains  a  technical  factor  of  Exfx:rience 
for  evaluation.  Can  only  the  technical  proposal  submitted  be  e\  aluated  and  considered? 
Can  the  experience  indicated  be  verified/researched? 

WhyAVhy  not? 

12.  Who  has  the  ultimate  authority  to  approve  the  Source  Selection  Plan  and  determine 
which  offeror  will  be  awarded  the  contract? 

13.  True  or  False:  The  offeror  receiving  the  highest  overall  (cost  and  technical)  evaluated 
score  automatically  is  selected  for  aw  ard  and  additional  evaluation  or  justification  is  not 
required. 

Why/why  not? 

14.  Explain  what  is  meant  by  offerors  being  technically  equivalent. 

15.  When  two  or  more  offerors  receive  essentially  the  same  technical  scores,  are  the\ 
always  technically  equivalent? 

Why/Why  not? 

16.  What  must  be  done  if  individual  committee  members'  technical  scores  of  an  offeror's 
proposal  vary  widely  on  one  or  more  technical  factors?  How  will  the  Committee’s 
summary  and  final  recommendation  be  affected? 

17.  The  Source  Selection  Plan  will  establish  technical  factors  to  be  evaluated  and  their 
weights.  Should  the  plan  and/or  the  committee  subdivide  by  weights  each  technical  factor? 

WhyAVhy  not? 

18.  The  plan  will  contain  a  technical  evaluation  rating  standard  (usually  numeric)  defining 
the  ratings  10  through  0.  How’  will  the  rating  standard  be  used  during  the  technical 
evaluation?  Also,  must  the  numeric  rating  assigned  by  supported  by  written 
documentation?  Explain  your  answer. 

19.  Assume  the  technical  evaluation  w  ill  be  on  four  technical  factors  to  be  evaluated  and 
the  offeror  being  evaluated  scores  a  perfect  score  in  three  of  the  four  factors  but  scores  a 
one  or  zero  on  the  fourth.  As  a  result,  the  overall  numeric  technical  score  indicates  a 
satisfactory  rating.  How  should  the  committee's  evaluation  and  final  recommendation 
address  this  situation? 

20.  Explain  the  relationship  between  the  evaluation  criteria  set  forth  in  the  Source 
Selection  Plan  and  the  technical  proposal  submission  requirement  set  forth  in  the 
solicitation. 

21 .  At  the  conclusion  of  technical  evaluations,  w'hat  documents  must  be  submitted  to  the 
PCO  by  the  Committee  Chairman. 
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is  included  in  contract  it  is  recommended  data  be  developed  cases  the  average  hourly  rales  from  the  offeror  will  group 

with  and  without  material  cost.  together  with  expected  differences  to  be  reflected  in  their 

If  the  offerors  are  providing  different  amounts  of  hours  the  "loadings." 

different  amounts  should  be  anayed. 
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